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Foreword 


The  primary  purpose  of  the  Image  Understanding  (IU)  Testbed  is  to  pro¬ 
vide  a  means  for  transferring  technology  from  the  DARPA-sponsored  IU 
research  program  to  DMA  and  other  organizations  in  the  defense  com¬ 
munity. 

The  approach  taken  to  achieve  this  purpose  has  two  components: 

(1)  The  establishment  of  a  uniform  environment  that  will  be  as  com¬ 
patible  as  possible  with  the  environments  of  research  centers  at 
universities  participating  in  the  IU  program.  Thus,  organizations 
obtaining  copies  of  the  Testbed  can  receive  a  flow  of  new  results 
derived  from  ongoing  research. 

(2)  The  acquisition,  integration,  testing,  and  evaluation  of  selected 
scene  analysis  techniques  that  represent  mature  examples  of  generic 
areas  of  research  activity.  These  contributions  from  participants  in 
the  IU  program  will  allow  organizations  with  Testbed  copies  to 
immediately  begin  investigating  potential  applications  of  IU  technol¬ 
ogy  to  problems  in  automated  cartography  and  other  areas  of  scene 
analysis. 

The  TU  Testbed  project  was  carried  out  under  DARPA  Contract  No. 
MDA903-79-C-0599.  The  views  and  conclusions  contained  in  this  document 
are  those  of  the  authors  and  should  not  be  interpreted  as  necessarily 
representing  the  official  policies,  either  expressed  or  implied,  of  the 
Defense  Advanced  Research  Projects  Agency  or  the  United  States  govern¬ 
ment. 

This  report  describes  the  RELAX  relaxation  package  contributed  by  the 
University  of  Maryland  and  presents  an  evaluation  of  its  characteristics 
and  features. 


Andrew  J.  Hanson 
Testbed  Coordinator 
Artificial  Intelligence  Center 
SRI  International 
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Abstract 


RELAX  is  a  system  of  routines  that  modifies  the  probabilities  associated 
•with  labels  attached  to  the  elements  of  a  two-dimensional  array.  These 
modifications  reflect  the  compatibility  of  each  element's  labels  with  those 
of  its  neighbors.  The  initial  probability  assignments  are  usually  derived 
from  local  property  values  in  the  neighborhood  of  each  pixel.  The  final 
assignments  may  be  used  for  object  detection  or  segmentation,  or  may 
be  mapped  back  to  image  intensities  to  achieve  noise  suppression, 
enhancement,  or  segmentation. 

The  relaxation  package  was  contributed  to  the  ARP  A/ DMA  Image  Under¬ 
standing  Testbed  at  SRI  by  the  University  of  Maryland.  This  report  sum¬ 
marizes  applications  for  which  RELAX  is  suited,  the  history  and  nature  of 
the  algorithm,  details  of  the  Testbed  implementation,  the  manner  in 
which  RELAX  is  invoked  and  controlled,  the  type  of  results  that  can  be 
expected,  and  suggestions  for  further  development. 
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Section  1 


Introduction 


The  RELAX  package  is  an  interactive  system  of  routines  for  mapping  digital  image  data 
into  a  probability  network,  modifying  the  probabilities  to  reflect  local  constraints,  and 
mapping  the  information  back  to  the  luminance  domain.  It  is  currently  configured  for 
image  enhancement  and  object  detection,  but  has  many  other  applications. 

Code  modules  and  test  data  for  the  RELAX  system  were  provided  by  the  Computer 
Vision  Laboratory  at  the  University  of  Maryland  (UM).  The  UM  relaxation  routines  are 
configured  as  a  set  of  stand-alone  programs  collectively  called  GPSPAR:  General- 
Purpose  Software  Package  for  Array  Relaxation.  This  package  was  originally  written  in 
the  C  language  by  Russel  C.  Smith  and  Joseph  A  Pallas  at  the  University  of  Maryland. 

The  current  RELAX  program  is  a  command  interpreter  that  interactively  invokes  the 
GPSPAR  routines.  This  version  of  RELAX  was  constructed  for  the  ARPA/DMA  Image 
Understanding  Testbed  at  SRI  International  by  Kenneth  Laws.  The  underlying  com- 
•  mand  interpreter  is  the  Cl  subroutine  provided  to  the  Testbed  by  Camegie-Mellon 
University  (CMU). 

Many  of  the  user-interface  and  image  access  routines  were  also  contributed  by  CMU. 
Particular  credit  is  due  to  Steven  Shafer  for  the  Cl  command  interpreter  and  related 
string  manipulation  routines,  to  David  Smith  for  the  image  access  software,  and  to 
David  McKeown,  assisted  by  Jerry  Denlinger,  Steve  Clark,  and  Joe  Mattis,  for  the  Grin- 
nell  display  software.  Kenneth  Laws  at  SRI  adapted  this  C-language  software  for 
Testbed  use  and  interfaced  it  with  the  University  of  Maryland  contributions. 

No  changes  were  required  in  the  relaxation  algorithm  itself.  The  information  in  this 
document  should  thus  be  considered  supplementary  to  the  material  cited  in  the  UM 
references. 

This  document  includes  both  a  user’s  guide  to  the  RELAX  system  and  an  evaluation  of 
the  algorithm.  Section  2  explains  the  nature  and  use  of  the  system  in  the  context  of 
typical  applications.  Section  3  surveys  the  historical  development  of  the  technique  and 
presents  the  current  algorithms  in  detail.  Section  4  describes  the  Testbed  implemen¬ 
tation  of  this  package  and  suggests  some  possible  improvements.  Section  5  instructs 
the  user  in  the  mechanics  of  using  the  RELAX  software.  Section  6  documents  the  per¬ 
formance  that  may  be  expected  in  various  circumstances  and  presents  the  results  of 
evaluation  tests.  Section  7  outlines  a  number  of  suggestions  for  improving  the  algo¬ 
rithm.  Section  8  is  a  very  brief  summary.  Appendix  A  shows  how  to  invoke  the  RELAX 
routines  in  the  manner  of  the  original  GPSPAR  package  submitted  by  UM. 
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Section  2 


Background 


This  section  presents  a  management  view  of  the  RELAX  program,  The  relaxation  algo¬ 
rithm  is  briefly  sketched.  Typical  applications  and  potential  applications  requiring 
further  development  of  the  algorithm  are  discussed,  and  related  applications  for  which 
other  algorithms  are  better  suited  are  noted. 


2.1.  General  Description 

The  RELAX  package  for  digital  image  enhancement  and  analysis  is  based  on  a  class  of 
algorithms  for  iteratively  modifying  vectors  of  probability  values  associated  with  the 
pixels  of  a  two-dimensional  array.  This  competitive-cooperative  relaxation  process 
strengthens  compatible  relationships  and  suppresses  incompatible  ones. 

The  RELAX  algorithms  also  illustrate  methods  of  propagating  global  interpretations 
and  constraints  through  a  network  by  local  updating  of  the  node  interpretations. 
Such  operations  show  promise  for  implementation  on  parallel  arrays  of  processors 
and  other  advanced  architectures. 

An  extensive  literature  connects  the  basic  relaxation  methods  with  numerous  appli¬ 
cation  areas.  Much  of  the  literature  discusses  relaxation  on  arbitrary  graph  struc¬ 
tures  rather  than  rectilinear  data  grids.  The  RELAX  package,  however,  is  aimed 
specifically  at  image-based  applications. 

The  user  starts  with  an  array  of  values  or  a  digital  image,  typically  a  luminance  image 
or  the  output  of  an  image-processing  operator.  The  value  at  each  pixel  (or  a  set  of 
values  from  a  neighborhood  of  each  pixel)  is  converted  to  a  vector  of  probabilities; 
each  probability  reflects  the  likelihood  that  the  pixel  should  be  assigned  a  particular 
semantic  label.  This  conversion  process  depends  on  the  user’s  goals  and  the  pixel 
classes  that  are  relevant  to  those  goals.  In  some  formulations  of  relaxation,  usually 
using  different  rules  for  adjusting  the  vectors  at  each  pixel,  the  vectors  can  be 
regarded  as  representing  fuzzy-set  memberships  [Zadeh65,  Kandel78]  rather  than 
probabilities.  In  this  report,  for  simplicity,  we  shall  regard  the  vectors  as  probability 
vectors  and  and  call  the  vector-valued  image  a  probability  image. 

Next  the  user  selects  a  relaxation  method,  a  neighborhood  size,  and  a  set  of  compati¬ 
bility  coefficients.  The  compatibility  coefficients  are  typically  generated  automati¬ 
cally  from  the  initial  probability  image.  This  will  be  discussed  in  more  detail  in  Sec¬ 
tions  3  and  5. 

The  user  then  initiates  one  or  more  "relaxation  steps.”  adjusting  the  probability  vec¬ 
tor  at  each  pixel  in  accordance  with  the  compatibility  relationships  and  the  probabil¬ 
ity  vectors  at  each  of  its  neighboring  pixels.  The  definition  of  "neighbor"  is  supplied 
by  the  user;  it  must  be  the  same  for  each  pixel. 

The  resulting  probability  vectors  are  typically  mapped  back  to  the  luminance  domain 
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Background 


so  that  the  user  may  observe  the  effect  of  the  relaxation.  This  is  not  strictly  neces¬ 
sary;  a  halting  criterion  based  on  the  probability  domain  may  be  employed.  The  final 
probability  image  may  or  may  not  be  mapped  back  to  a  luminance  image,  depending 
on  the  needs  of  the  user. 

Relaxation  is  a  philosophically  attractive  procedure  that  seeks  a  globally  consistent 
interpretation  through  local  processing.  Relaxation  is  still  in  the  early  stages  of 
development  and  needs  further  research  to  determine  the  nature  and  range  of  its 
future  applications. 


2.2.  Typical  Applications 

The  RELAX  program  may  be  used  in  any  application  requiring  noise  suppression  or 
feature  reinforcement.  The  results  of  an  image  operation,  such  as  edge  detection, 
can  be  “smoothed”  and  detection  reinforced.  These  effects,  useful  in  themselves, 
may  be  precursors  to  further  analysis. 

The  RELAX  package  is  primarily  adapted  to  specific  applications  by  the  mapping 
functions  that  convert  luminance  images  to  probability  images.  Very  few  such  map¬ 
pings  are  currently  available  with  the  package.  Those  that  have  been  provided  are 
suitable  for  the  following  purposes: 

*  Requantization— Reduction  of  the  number  of  gray  levels  in  an  image  typi¬ 
cally  introduces  visible  false  edges  in  areas  of  smooth  gradient.  Relaxation 
may  be  used  to  pull  pixels  near  the  quantization  threshold  into  the  next 
higher  or  lower  gray  level.  This  will  reduce  false  contours  and  act  as  a  seg¬ 
mentation  technique  if  the  relaxation  tends  to  group  pixels  that  are  within 
the  same  imaged  object.  (Adding  a  random  dither  signal  to  the  image 
prior  to  requantization  would  also  reduce  false  contouring,  but  would 
degrade  the  image  and  any  subsequent  segmentation  of  it.) 

*  Histogram  Sharpening -The  re  have  been  several  schemes  for  iteratively 
replacing  pixel  values  by  some  function  of  neighboring  values  in  order  to 
sharpen  the  peaks  of  the  image  histogram  [Rosenfeld78,  Peleg78b, 
Bhanu82].  Repeated  applications  can  be  used  to  merge  smaller  histogram 
peaks  into  larger  ones  until  only  a  set  number  remain.  (This  differs  from 
requantization  in  that  the  resulting  gray  levels  need  not  be  equally 
spaced.)  Histogram  sharpening  is  sometimes  used  as  a  precursor  to  image 
segmentation  or  compression. 

*  Smoothing— Relaxation  can  be  used  to  smooth  image  regions  to  reduce 
noise  artifacts.  The  smoothing  can  be  done  without  blurring  region  edges 
if  adjacent  regions  are  mapped  fairly  well  into  different  a  priori,  labels. 
(Edge -preserving  smoothing  without  such  conditions  requires  special- 
purpose  techniques  that  test  region  homogeneity  before  applying  the  com¬ 
patibility  correction.  A  median  filter  works  this  way.) 

*  Edge  Enhancement— Relaxation  can  be  used  to  sharpen  region  boundaries 
while  smoothing  the  interiors.  (Here,  too,  special-purpose  algorithms  that 
include  decision  logic  might  have  better  success  than  a  linear  summation 
of  compatibility  constraints.)  Relaxation  can  also  be  applied  to  a  gradient 
image  to  enhance  extended  discontinuities  and  suppress  noise  spikes. 
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*  Linear-Feature  Enhancement— This  is  essentially  the  same  as  edge 
enhancement,  although  more  sophisticated  feature  detectors  and 
classification  operators  might  be  involved. 

*  Detection— It  is  a  small  step  from  enhancing  a  feature  to  detecting  it. 
Relaxation  can  help  by  reinforcing  detection  of  groups  of  similar  pixels 
while  suppressing  detection  of  isolated  noise  points.  Other  methods  may 
be  more  advantageous  for  detecting  objects  larger  or  smaller  than  a  few 
pixels. 

*  Pixel  Classification— The  source  class  to  which  a  pixel  is  assigned  may  be 
adjusted  by  using  the  classifications  and  arrangement  of  its  neighbors. 
Iteration  of  this  process  can  reduce  the  effect  of  texture  on  classification. 
The  RELAX  package  supplied  by  the  University  of  Maryland  contained  a 
demonstration  of  two-class  segmentation  by  thresholding,  using  an 
infrared  image  of  a  military  tank.  Segmentation  by  pixel  classification  into 
multiple  classes  using  relaxation  is  described  by  Eklundh  et  al. 
[EklundhBO].  The  anomaly  detection  experiments  documented  in  Section  6 
are  also  related  to  pixel  classification 


2.3.  Potential  Extensions 

The  following  applications  might  be  feasible  if  the  RELAX  package  were  modified,  used 
in  a  nonstandard  fashion,  or  integrated  into  a  more  sophisticated  system 


*  Clustering— Clusters  of  points  in  a  metric  space  can  be  detected  by  allowing 
each  point  to  "gravitate”  toward  its  neighbors.  This  is  the  spatial  analogue 
of  histogram  sharpening.  It  requires  a  graph-based  relaxation  algorithm 
instead  of  the  image-based  RELAX  updating  algorithm. 

*  Semantic  Labeling— Relaxation  can  be  used  to  derive  consistent  sets  of 
names  or  interpretations  for  regions  in  a  scene.  This  also  requires  a 
graph-based  relaxation  method.  A  similar  application  is  the  identification 
of  mixed  pixels,  noise  regions,  and  border  slivers  in  segmented  images. 

Such  applications  have  been  described  in  numerous  papers,  and  there  have  been 
numerous  other  applications  of  relaxation  techniques  to  image  processing 
[Rosenfeld77b,  Rosenfeld82,  Rosenfeld83], 


2.4.  Alternative  Approaches 

This  section  describes  applications  that  are  similar  to  RELAX  applications,  but  which 
differ  in  some  fundamental  fashion.  While  the  difficulties  with  applying  RELAX  might 
be  overcome,  other  techniques  would  often  be  more  appropriate. 


*  Noise  Suppression— Despite  the  applicability  of  relaxation  to  smoothing,  the 
updating  algorithms  in  the  RELAX  package  have  no  underlying  model  of 
image  and  noise  characteristics.  Image  noise  can  be  more  effectively 
removed  by  filtering  techniques  based  on  the  noise  statistics. 
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*  Restoration— Similarly,  blur  and  other  degradations  are  best  removed  by 
techniques  that  model  the  degradation  process.  The  RELAX  package  can 
be  used  for  limited  classes  of  image  enhancement,  but  usually  at  the  cost 
of  introducing  less  visible  degradations  elsewhere. 

As  a  rule,  relaxation  methods  work  best  in  those  applications  requiring  "gravita¬ 
tional''  or  "fluid  flow”  solutions,  such  as  histogram  sharpening  or  image  smoothing. 
They  might  also  be  useful  for  enhancement  applications  that  can  be  cast  as  "reverse 
fluid  flow"  problems.  They  generally  do  not  work  as  well  as  model-based  restoration 
or  analysis  methods  when  there  exist  underlying  models  of  the  scene  and  the  image 
formation  process. 
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Description 


This  section  presents  the  history  of  relaxation  for  image  processing  and  a  detailed 
statement  of  the  algorithms  used  in  the  RELAX  package.  The  historical  information  is 
intended  to  clarify  the  major  issues  in  iterative  image  processing  and  to  guide  the 
reader  to  the  relevant  literature. 


3.1.  Historical  Development 

Relaxation  methods  are  iterative  procedures  designed  to  seek  adequate  solutions  to 
problems  that  defy  analytic  analysis.  Relaxation  advances  toward  a  solution  state 
( e.g .,  a  fully  segmented  image)  step  by  step,  instead  of  solving  for  the  optimal  fixed- 
point  solution  (as  is  common  in  the  image  restoration  literature).  Either  the  user 
stops  the  process  or  an  automatic  halting  criterion  is  invoked  when  the  remaining 
.  errors  are  sufficiently  small.  The  operation  is  similar  to  hill  climbing,  combinatorial 
optimization  [Kirkpatrick83] ,  or  stochastic  approximation  approaches. 

Relaxation  methods  have  long  been  used  in  physics  and  engineering,  particularly  in 
computational  fluid  dynamics,  aerodynamics,  and  thermodynamics.  Systems  of  ordi¬ 
nary  and  partial  differential  equations  are  commonly  solved  by  approximate  numeri¬ 
cal  methods.  Some  finite-element  techniques  propagate  local  constraints  through  a 
field  in  a  single  pass;  other  techniques  are  iterative. 

Relaxation  techniques  may  have  entered  the  image-understanding  literature  through 
constraint  satisfaction  networks  used  to  label  line  drawings  [Guzman68,  Waltz72, 
Haralick79],  The  early  procedures  assumed  all  possible  labelings  for  each  adjacent 
line  pair  in  the  scene,  then  eliminated  incompatible  label  pairs.  Convergence  was 
very  rapid,  but  these  methods  had  no  mechanism  for  handling  probabilities  or  uncer¬ 
tain  evidence. 

Constraint  networks  were  later  generalized  to  many  image-understanding  and 
expert-system  applications  [Montanari74,  Hart77,  Rosenfeld77b],  particularly  to  the 
consistent  labeling  of  scene  regions  [Yakimovsky73,  Barrow76,  Freuder76,  Fau- 
gerasSl],  This  movement  merged  with  the  development  of  iterative  techniques  for 
texture  segmentation  and  identification  [Troy73],  image  region  growing  and  merging 
[Brice70,  Yakimovsky76,  Zucker76],  image  smoothing  and  enhancement  [Davis77a, 
Lev77],  histogram  modification  [Rosenfeld77a],  edge  detection  [Eberlein76, 
Schachter76]  linear-feature  enhancement  [Riseman77,  Zucker77,  VanderBrug77], 
curve  segmentation  [Davis77b],  shape  matching  [Davis77c],  and  other  applications. 

The  result,  generally  called  relaxation  labeling,  is  a  set  of  iterative  probabilistic 
approaches  that  consolidate  many  applications  in  the  above  areas.  Probabilistic 
labeling,  continuous  relaxation,  and  stochastic  labeling  are  other  names  for  these 
techniques.  All  involve  the  application  of  local  constraints  to  vary  the  weights  (or 
degrees  of  belief)  of  semantic  labels  attached  to  the  nodes  of  a  graph.  Iteration  of 
the  local  adjustments,  either  in  sequence  or  in  parallel,  are  presumed  to  drive  the 
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network  closer  to  a  global  optimum  or  to  a  fixed  point  of  the  relaxation  (i.e.,  a  state 
unaffected  by  further  iterations). 

The  original  relaxation  labeling  algorithm  was  proposed  by  Hummel,  Zucker,  and 
Rosenfeld  at  the  University  of  Maryland  [Rosenfeld76,  Hummel78]  and  was  further 
developed  by  a  number  of  other  researchers  [Peleg78a,  Zucker7Ba,  Zucker7Bb, 
Yamamoto79,  HaraiickBD,  HummelBQ,  O'LearyBO].  This  method  makes  use  of  additive 
updating  formulas.  A  later  approach,  based  on  multiplicative  updating,  was  also 
developed  at  the  University  of  Maryland  [KirbyBO,  PelegBOa]. 

Many  researchers  have  offered  evaluation  and  discussion  of  the  Umits  of  relaxation 
processing  [KirbyBO,  KitchenSC,  O'LearyBO,  Richards60,  FeketeBl,  Diamona82, 
Nagin82,  HaralickB3].  Faugeras  and  Berthod  have  suggested  an  alternative  relaxa¬ 
tion  labeling  philosophy  [FaugerasBOa,  FaugerasBOb],  and  this  in  turn  has  been  criti¬ 
cized  [HummelBO], 

The  papers  cited  above  generally  discuss  relaxation  in  the  abstract,  although 
numerous  applications  have  been  developed  [RosenfeldB2,  RosenfeldB3].  The  original 
papers  on  the  Hummel-Zucker-Rosenfeld  and  Peleg  methods  [Rosenfeld76,  PelegBOa] 
still  constitute  a  good  introduction  to  the  philosophy  of  relaxation. 


3.2.  Algorithm  Description 

« 

While  the  RELAX  program  provides  a  stand-alone  implementation  of  relaxation,  there 
are  other  ways  of  implementing  it.  Often  the  relaxation  algorithm  is  so  integrated 
with  other  techniques  that  it  would  be  difficult  to  isolate  the  "relaxation  part"  of  a 
procedure.  Relaxation  is  a  philosophy;  no  one  algorithm  embodies  its  alternate  for¬ 
mulations.  The  two  procedures  included  in  the  RELAX  program  (Hummel-Zucker- 
Rosenfeld,  or  HZR,  and  Peleg)  are  representative  of  the  algorithms  used  in  most 
applications  of  relaxation. 


3.2.1.  General  Approach 

Most  image-based  relaxation  procedures  comprise  the  four  stages  listed  below;  in 
rare  circumstances  one  of  the  stages  may  be  skipped.  The  stages  are  essentially 
input,  construction  of  the  relaxation  operator,  relaxation  per  se,  and  output.  Each 
stage  significantly  influences  the  effectiveness  of  the  overall  relaxation  procedure. 

The  four  stages  of  the  relaxation  process  are  as  follows: 

*  Image-to-PrababUity  Mapping 

An  operator  is  applied  to  the  image  array  to  convert  the  luminance 
values  (or  local  property  values,  eic.)  to  probability  vectors.  Each  pixel 
is  assigned  a  vector  representing  an  arbitrary  set  of  semantic  labels. 
Numeric  values  of  the  vector  elements,  may  be  probabilities,  likelihoods, 
or  other  measures  of  belief  in  the  applicability  of  the  corresponding 
labels  at  that  image  point.  It  is  this  mapping  that  adapts  the  relaxation 
paradigm  to  a  specific  application.  The  RELAX  package  currently  con¬ 
tains  illustrative  mapping  functions  for  image  smoothing  and  edge  detec¬ 
tion  applications.  The  user  will  generally  have  to  supply  new  mapping 
routines  for  these  or  other  applications. 


7 


Description 


*  Compatib ility  Computation 

Coefficients  of  compatibility  between  labels  on  neighboring  pixels  are 
computed,  usually,  from  the  weighted  frequencies  of  label  adjacencies  in 
the  original  probability  image.  These  coefficients  essentially  define  the 
.  local  relaxation  operations  that  will  be  applied  to  the  probability  image. 
The  RELAX  package  contains  one  routine  for  estimating  HZR  coefficients 
and  another  for  Peleg  coefficients;  the  values  may  also  be  supplied  manu¬ 
ally. 

*  Relaxation  Updating 

The  compatibility  coefficients  and  updating  rule  are  used  to  modify  the 
probability  vector  at  each  pixel  in  turn.  The  values  in  each  vector  are 
adjusted  by  a  weighted  sum  or  product  of  values  at  neighboring  pixels  to 
enhance  compatible  combinations  and  suppress  incompatible  ones.  The 
user  may  call  for  one  or  more  passes  of  the  relaxation  operator  through 
the  probability  image;  current  methods  do  not  have  halting  criteria  to 
determine  when  the  updating  should  terminate. 

*  Pmbability-to-Image  Mapping 

Relaxation  methods  may  be  designed  to  derive  one  or  more  numbers  per 
pixel;  these  may  be  image  luminances,  edge  probabilities,  object  likeli¬ 
hoods,  segmentation  maps,  or  other  interpretations.  The  user  must  sup¬ 
ply  a  mapping  procedure  to  convert  the  multivariate  probability  image 
to  this  form.  Routines  currently  in  the  RELAX  package  can  produce 
binary  and  gray-level  luminance  images  useful  for  edge  or  object  detec¬ 
tion  as  well  as  for  image  enhancement. 

We  shall  now  present  these  stages  in  detail. 


3.2.2.  Image-to-Probability  Mapping 

Luminance  images  used  as  input  to  a  relaxation  procedure  must  first  be  converted 
into  probability  image  form.  The  method  supplied  in  the  RELAX  package  imgprb 
command  is  described  here.  It  is  a  linear  mapping  of  image  brightness  to  a  vector 
of  probability  values  representing  our  belief  that  the  pixel  belongs  in  particular 
luminance  "level  slices."  This  mapping  might  be  suitable  for  image  binarization  or 
requantization,  noise  suppression,  and  object  detection  applications.  It  is  provided 
only  as  an  example  and  as  a  mechanism  for  verifying  the  correct  operation  of  the 
relaxation  updating  software;  other  mapping  functions  will  be  needed  for  other 
applications. 

Among  the  arguments  to  imgprb  are  the  low  and  high  gray-level  values  and  the 
number  of  labels  to  use.  Pixel  values  are  clipped  to  lie  within  the  specified  range 
and  are  then  mapped  to  vectors  of  probability  values  between  0.0  and  1.0.  The 
mapping  function  depends  on  the  number  of  labels,  as  discussed  below. 

For  binary  output,  the  label  probabilities  may  be  thought  of  as  positive  and  nega¬ 
tive  support  for  the  hypothesis  that  a  bright  object  is  the  pixel  source.  These  are 
represented  by  two  floating-point  numbers  per  pixel,  p[0]  and  p[l],  where  the 
indices  represent  the  two  classes  or  pixel  labels.  The  first  value,  p|_0],  represents 
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our  belief  that  the  pixel  is  from  a  light  "object"  area;  the  second,  p[l]=1.0-p[0], 
our  belief  that  it  is  from  a  dark  background  population.  The  lowest  pixel  value  in 
the  specified  range  thus  maps  to  p[0]=0.0  andp[l]=1.0;  the  highest  to  jd[0]=1.0 
and  p  [  1]=0.0.  Linear  interpolation  is  used  for  intermediate  pixel  values. 

If  the  user  requests  more  than  two  labels,  the  labels  may  be  thought  of  as  level 
slices  and  the  vector  elements  as  probabilities  that  a  pixel  should  be  assigned  to 
one  of  these  levels.  The  lowest  pixel  value  maps  to  p  [0]=1.0,  the  highest  to 
p[maximum  iobei]=1.0.  Linear  interpolation  between  the  bracketing  labels  is  used 
for  intermediate  pixel  values.  At  most  two  labels  will  have  nonzero  probabilities 
with  this  conversion  scheme. 


3.2.3.  Compatibility  Computation 

A  compatibility  coefficient  must  be  specified  for  each  combination  of  a  label  at  the 
central  pixel  with  each  label  at  each  neighboring  pixel.  The  values  of  the 
coefficients  depend  on  the  updating  method  to  be  used  and  also,  as  a  rule,  on  the 
initial  probability  image  data. 

The  neighborhood  of  a  pixel  is  usually  taken  to  be  the  set  of  pixels  in  a  small  sur¬ 
rounding  square,  with  the  size  of  the  square  selectable  by  the  user.  Neighborhoods 
restricted  to  only  horizontal  or  vertical  neighbors  or  that  include  nonadjacent  pix¬ 
els  are  sometimes  required;  the  defn.br  routine  allows  such  arbitrary  neighbor¬ 
hoods  to  be  defined. 

Compatibility  coefficients  used  for  the  Hummel-Zucker-Rosenfeld  relaxation 
scheme  are  different  from  those  used  for  its  Peleg  counterpart.  In  the  additive 
HZR  scheme  [Hummel7B,  Peleg7Ba,  Yamamoto79].  coefficients  are  negative  if  the 
labels  are  incompatible,  zero  if  they  are  independent,  and  positive  if  the  labels  are 
compatible.  Coefficient  values  are  restricted  to  the  range  [-1.0,  +1.0].  They  typi¬ 
cally  range  from  -0.1  to  about  0.5  when  computed  with  the  RELAX  he ampat  routine. 

In  the  multiplicative  Peleg  scheme  [Peleg80a,  Haralick83],  all  the  coefficients  are 
nonnegative.  Coefficients  are  less  than  unity  if  the  labels  are  incompatible,  unity  if 
they  are  independent,  and  greater  than  unity  if  the  labels  are  compatible.  They 
typically  range  from  near  zero  to  about  5.0  when  computed  with  the  RELAX  pcam.- 
pat  routine. 

Compatibility  is  a  loosely  defined  term,  and  no  definition  to  date  has  been  entirely 
satisfactory  [HaralickB3J.  The  HZR  compatibility  coefficients  are  based  on  informa¬ 
tion  theory  [Peleg78a,  Yamamoto79],  the  corresponding  Peleg  coefficients  on  con¬ 
ditional  probabilities  [PelegBOa],  Both  methods  utilize  the  a  priori  probabilities  of 
labels  at  an  arbitrary  pixel,  as  measured  in  the  initial  probability  image,  to  esti¬ 
mate  the  compatibilities. 

Suppose  we  have  a  graph  whose  nodes  are  each  to  be  labeled  with  one  of  the  possi¬ 
ble  labels  Ax,  Ag . A* . A„,  Further  suppose  that  some  measurement  asso¬ 

ciated  with  each  node  has  allowed  us  to  state  the  a  priori  probability  of  that  node 
being  labeled  with  each  of  the  possible  labels.  For  node  i  we  have  Pt(At), 
fc=l,  •  •  •  ,n,  as  the  probability  that  node  i  has  label  A*. 

If  we  were  to  label  the  ith  node  with  the  label  A*,  and  if  the  graph  showed  that  the 
ith  node  and  the  jth  node  are  adjacent,  then  we  need  to  determine  whether  the 
label  Ajt  on  the  ith  node  is  compatible  with  the  labels  on  the  j'th  node.  We  specify 
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this  compatibility  with  the  coefficient  which  states  the  compatibility  of 

label  A*  on.  the  ith  node  with  the  label  A*  on  the  jth  node. 

For  the  HZR  scheme,  compatibilities  may  be  calculated  from  the  quantity 


*V(**,Ai)  =  i-ln 


i  i 


fc=  1.  ' 
1=1. 


,n 

,n 


where  i  ranges  over  all  w  nodes  of  the  graph  and  j  specifies  the  particular  neigh¬ 
bor  of  the  ith  node.  For  each  node  i ,  the  compatibility  with  its  j  th  neighbor  is  then 


ry(^b  = 


-1  if  (Aji-.Ai)  <  — 1 

rj(  A*,Ai)  if-l^*’J(AfclA|)s  +1 

+  1  if  +1  <rj(A*,Ai) 


Note  that  the  RELAX  package  currently  uses  the  same  values  of  the  compatibility 
coefficients  for  ail  values  of  i,  i.e.,  for  each  pixel  postion  to  be  updated. 

For  the  Peleg  scheme,  the  compatibilities  may  be  calculated  from  the  quantity 


i  i 


k= 1,  *  •  •  ,n 
l  =  1.  ■  •  •  ,n 


where  i  ranges  over  all  w  nodes  of  the  graph  and  j  specifies  the  particular  neigh¬ 
bor  of  the  ith  node.  For  each  node  i,  the  compatibility  with  its  j  th  neighbor  is  then 


rij  *\)  -  rj  i^l ) 


Note  that  these  Peleg  compatibilities  are  in  the  range  [O.-uj]. 

It  may  sometimes  be  desirable  to  use  ensemble  statistics  to  compute  the  compati¬ 
bilities.  Only  experience  with  a  particular  application  allows  coefficients  to  be 
chosen  rather  than  calculated  by  formula. 


3.2.4.  Updating  Formulas 

The  relaxation  updating  computations  can  now  be  presented  in  more  detail. 

The  goal  of  the  relaxation  algorithm  is  to  update  the  values  of  the  probabilities 
associated  with  a  node  so  as  to  take  into  account  the  compatibility  of  neighboring 
labels.  A  number  of  different  schemes  have  been  proposed  to  do  this  updating.  The 
earliest  was  the  HZR  scheme  [Rosenfeld76],  in  which  the  (t  +  1)  update  of  the  proba¬ 
bility  values  is  calculated  from  the  (f )  values  by  the  following  rule: 
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For  node  i. 


Pi{  K*±)  =  -r - 

gPi^(X)[l+g/£)(A)] 


k- 1,  •  •  •  ,n 


£  ’•«(X».X')pJ(0(X') 

*’=*! 


fc  =  l,  •  ■  •  ,71 


where  j  indexes  the  m  neighbors  of  node  i,  and  r„(Ajk,\')  is  the  compatibility 
coefficient  for  node  i  with  label  A*  and  neighboring  node  j  with  label  A'. 


The  Peleg  relaxation  scheme  [PelegBOa],  also  included  in  this  package,  uses  the 
updating  rule 


prnM  -  jjr-E 

m  i 


R(0(\b)g</°(\b) 
£  Ptl0W  W°(X) 


k  =  l,  ■  ■  ■  ,n 


gj0(xt)  =  ^  ’ry(xt.x,)p/‘)(x')  fc=i,  •  •  ■  ,7i 

v=xl 

where  ji  indexes  the  m  neighbors  of  node  i.  (Actually,  a  slightly  more  general  form 
of  the  Peleg  scheme  is  implemented  in  the  RELAX  system;  see  Section  5.2  for 
details.) 


In  both  of  these  schemes,  gi(X*)  (or  (X*))  can  be  thought  of  as  the  neighboring 
node’s  assessment  (by  node  j  in  the  case  of  giy(Xjt))  that  node  i  should  be  labeled 
X t ;  Pi(Xfc)  is  the  assessment  by  node  i  that  its  own  label  should  be  A*.  These  two 
assessments  are  combined  to  produce  an  updated  probability. 


Other  forms  of  the  updating  rule  based  on  optimizing  measures  that  are  functions 
of  terms  like  the  above  p's  and  g's,  have  been  employed  [FaugerasBOa, 
FaugerasBOb,  HummelBO].  While  the  development  of  these  forms  rests  on  a  firmer 
foundation  than  that  of  the  rules  above,  these  newer  rules  also  have  defects 
[O'LearyBO,  PelegBOb].  Substantial  theoretical  work  needs  to  be  done  to 
comprehend  the  nature  of  relaxation  and  to  lay  the  groundwork  for  eliminating 
rule  deficiencies  in  the  future. 


The  foregoing  description  has  been  couched  in  terms  of  the  probability,  likelihood, 
certainty,  or  favorability  of  assigning  a  particular  label  to  a  node.  It  is  useful  to 
think  of  the  associated  numeric  value  as  the  probability  that  the  node  has  the 
label,  but  there  are  theoretical  problems  with  this  interpretation.  We  shall  adopt 
the  convenience  of  referring  to  such  values  as  probabilities— but  it  should  be  kept  in 
mind  that,  strictly  speaking,  this  may  not  be  correct. 

Relaxation  is  an  updating  rule  for  improving  the  initial  assignment  of  labels  by 
enforcing  compatibility  with  neighboring  labels.  Unfortunately,  compatibility  is  a 
poorly-understood  notion.  One  does  not  know  when  a  rule  will  converge,  or,  if  it 
does,  what  its  rate  of  convergence  will  be  [Zucker7Ba].  Nevertheless,  relaxation 
has  been  successfully  applied  to  a  number  of  different  problems.  Relaxation 
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captures  the  ideas  that  we  should  be  able  to  label  objects  so  that  they  are  compati¬ 
ble  with  their  neighbors,  and  that  we  should  be  able  to  do  this  through  local  pro¬ 
cessing  [Ullman79].  Relaxation  requires  a  solid  theoretical  base  [Hummel78,  Hum- 
melBO  J  to  define  its  domain  of  applicability. 


3.2.5.  Probability-to-Image  Happing 

When  the  relaxation  system  is  used  for  object  cueing,  the  matrix  of  p[0]  values  may 
be  the  desired  output.  In  other  cases,  it  may  be  desirable  to  map  the  probability 
vectors  back  to  luminance  gray  levels  so  that  the  output  can  be  displayed.  This 
mapping  is  typically  the  inverse  of  that  used  to  convert  a  luminance  image  to  pro¬ 
bability  form.  The  RELAX  package  prbimg  routine  is  the  inverse  of  the  imgprb  map¬ 
ping  described  earlier. 

Inversion  of  the  image-to-probability  mapping  is  complicated  by  the  fact  that  the 
"probability'*  vectors  for  a  pixel  are  not  always  normalized  and  need  not  sum  to 
1.0.  The  inverse  mapping  function  supplied  by  the  University  of  Maryland  uses 
different  resolutions  of  this  problem  in  the  two-label  and  multilabel  cases. 

The  prbimg  routine  will  convert  a  two-label  probability  image  to  a  gray-scale  image 
whose  values  quantify  the  "strength  of  belief"  in  the  p[0]  hypothesis  represented 
by  the  stronger  of  the  two  probabilities.  Thus,  p[0]  values  are  converted  directly  to 
pixel  values;  p[l]  values,  if  stronger,  are  subtracted  from  1.0  before  conversion  to 
pixel  values.  The  gray-level  interpolation  inherent  in  this  procedure  produces  an 
image  quantized  to’ an  arbitrary  number  of  gray  levels,  usually  256. 

A  multilabel  probability  image  will  be  converted  to  a  gray-scale  image  with  values 
representing  the  label,  or  luminance-level  slice,  with  the  highest  probability.  Thus, 
a  strongest  p  [maximum.  labei\  maps  to  the  highest  pixel  value  and  a  strongest  p[0] 
maps  to  the  lowest;  intermediate  labels  map  to  intermediate  gray  levels.  There  is 
no  interpolation  between  gray  levels,  so  the  output  image  is  quantized  to  the 
number  of  labels  used. 


Section  4 


Implementation 


This  section  documents  the  SRI  Testbed  implementation  of  RELAX.  It  is  intended  as  a 
guide  for  system  maintainers  and  for  programmers  modifying  the  RELAX  system.  The 
terms  used  in  this  section  are  either  defined  elsewhere  in  this  report  or  come  from  the 
supporting  operating  systems.  The  SRI  Testbed  uses  the  EUNICE  operating  system, 
which  is  a  Berkeley  UNIX1  emulator  for  VAX  computers  using  DEC'S  VMS  operating  sys¬ 
tem.  All  of  the  relaxation  software  will  also  run  on  a  pure  UNIX  system. 

The  RELAX  package  is  currently  compiled  as  an  interactive  driver  program.  The  driver 
invokes  other  programs  for  most  of  its  work,  although  it  does  call  subroutines  directly 
to  enable  conversion  between  intensity  and  probability  formats.  The  computational 
algorithm  is  very  little  changed  from  the  original  University  of  Maryland  version,  but 
the  command  interpreter,  help  system,  and  supporting  image-access  and  display  utili¬ 
ties  are  all  new. 

The  main  program  and  related  files  are  in  directory  /in/tb/src /relax.  Major  subdirec¬ 
tories  are 


cvlhdrlib 

defcom 

defnJbr 

demo 

help 

imgprb 

prbimg 

relax 

relaxpar 

src  /hummel 

src/peleg 


-  probability  image  subroutines; 

-  defcom  source  code; 

-  defnbr  source  code; 

-  shell  script  for  the  tank  demo; 

-  help  system  text  files; 

-  imgprb  source  code; 

-prbimg  source  code; 

-  relax  main  program  source  code; 
-relaxpar  source  code  (used  by  setup); 

-  hcompat  and  hrelax  source  code; 

- pcompat  a.nd  pretax  source  code. 


Compiled  versions  of  these  main  programs  are  kept  in  /iu/tb/bin.  The  Hummel  and 
Peleg  operators  are  not  kept  in  compiled  form,  since  they  are  intended  to  be  custom¬ 
ized  for  each  application. 

CuUuJrlib  is  a  library  of  subroutines  for  manipulating  the  floating-point  probability  files. 
(It  is  expected  that  someday  this  function  will  be  absorbed  by  the  Testbed  image¬ 
accessing  code.) 

The  imgprb  and  prbimg  programs  simply  parse  their  command-line  arguments  and 
pass  the  information  to  subroutines.  The  RELAX  driver  likewise  parses  commands 
given  to  it  and  invokes  the  same  subroutines.  These  subroutines  are  currently  stored 
in  directory  /iu/tbAib  /uisionlib,  specifically  in  the  relaxlib  subdirectory. 


‘UNIX  is  a  trademark  of  Bell  Laboratories. 
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Source  code  and  help  files  for  the  Cl  driver  are  in  / vu/tb/Hb/cUib .  For  extensive  docu¬ 
mentation,  type  "man  ci"  or  run  "vtroff  -man  /iu/tb/man/man3/ci.3c‘\  The  Cl  driver 
uses  command-line  parsing  routines  in  cilib /cmuarglib  and  in 
Au/tbAvb/sublib/askliJb;  both  of  these  may  someday  be  replaced  by  the  Testbed 
argument-parsing  routines  in  sublib  /arglib. 

Other  utility  routines  contributed  by  CMU  have  been  distributed  to 
AuAbAib/dsplib/gmrlib,  /iu. /tb  /lib  Amglib ,  and  Au/tb  Aib  /sublib ,  and  are  docu¬ 
mented  for  the  man  system  in  AuAb  /man/man3.  Some  of  these  have  been  modified 
or  rewritten  for  the  Testbed  environment;  the  image  access  code,  for  instance,  reads 
Testbed  image  headers  in  addition  to  CMU  image  headers.  Other  routines  in  these 
directories  were  developed  at  SRI. 

To  recompile  one  of  the  relaxation  routines,  e.g.,  imgprb,  just  connect  to  the  appropri¬ 
ate  source  directory  and  type  "make”.  You  may  type  "make  -n"  to  see  what  will  hap¬ 
pen  if  you  do  this.  Additional  options  are  documented  in  the  header  of  the  makefile. 
To  compile  and  install  the  entire  system,  run  the  make  program  in  Au/tb /src /relax. 
For  more  flexible  maintenance  options,  see  the  header  sections  of  the  corresponding 
makefile  files. 

The  hcompat  and  hrelax  programs  and  their  Peleg  equivalents  are  normally  compiled 
interactively  by  using  the  setup  command  of  the  RELAX  package.  Implementation  of 
this  capability  requires  that  the  source  file  locations  of  these  files  be  known  to  the 
.RELAX  program.  This  program  must  therefore  be  modified  and  recompiled  any  time 
the  relaxation  source  files  are  moved. 

RELAX  demonstrations  have  been  set  up  in  subdirectories  relax  and  tank  of 
Au/testbed/demo.  Just  connect  to  the  appropriate  directory  and  run  the  demo  com¬ 
mand.  Afterwards  you  may  want  to  run  the  cleanup  script  stored  in  that  directory  to 
delete  the  relaxation  output  files. 

The  UM  code  represents  an  interesting  style  of  programming  and  usage.  Two  points 
should  be  noted: 

*  Each  routine  is  compiled  as  a  separate  program.  Commands  are  passed  to  a 
command  interpreter  or  to  the  UNIX  shell  to  invoke  the  programs  in  the 
proper  sequence.  One  benefit  is  that  any  routine  can  be  altered  without 
recompiling  and  linking  the  entire  system.  (Another  possibility,  not  imple¬ 
mented  here,  is  to  pipe  the  routines  together  so  that  each  feeds  its  output  to 
the  next.  This  would  eliminate  many  of  the  intermediate  files  now  being 
created  by  the  system.) 

*  One  of  the  steps  in  a  relaxation  sequence  can  be  the  construction  and  compi¬ 
lation  of  a  special-purpose  relaxation  operation.  This  is  currently  done  by 
the  setup  routine,  which  uses  am  include  file  built  by  relaxpar  to  tailor  the 
hcompat  and  hrelax  programs  (or  their  Peleg  equivalents)  to  the  desired 
neighborhood  definition.  Such  operators  cam  run  faster  than  general- 
purpose  ones  that  use  run-time  evaluation  of  conditionals  to  control  execu¬ 
tion  logic. 

We  have  attempted  to  retain  this  style  of  programming  while  still  packaging  the  relaxa¬ 
tion  system  in  a  form  similar  to  that  of  other  major  software  systems  on  the  Testbed. 
We  have  retained  the  concept  of  separate  compilation  so  that  the  shell  script  in  Appen¬ 
dix  A  will  execute  properly  on  the  Testbed.  Those  who  prefer  such  a  system  are  free  to 
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make  use  of  it. 

We  have  also  'written  an  interactive  driver  package,  known  as  RELAX,  for  invoking  the 
routines  in  a  more  structured  environment;  this  allows  for  easy  incorporation  of  cus¬ 
tomized  syntax,  argument  defaulting,  global  status  variables,  help  functions,  and  other 
"intelligent"  session  control  features.  (Very  few  such  features  are  now  implemented, 
but  the  possibilities  cam  be  seen  in  other  Testbed  software  using  the  Cl  driver  mechan¬ 
ism.) 

Various  problems  were  encountered  in  integrating  the  original  package  with  the  IU 
Testbed  and  in  documenting  the  result.  Sometimes  it  was  easier  to  change  the  user 
interface  slightly  than  to  document  an  inconsistency.  Among  the  changes  made  in  the 
original  system  are  the  following: 


*  Name  Changes 

The  program  to  convert  images  to  a  probability  format  was  originally  named 
init,  and  the  program  to  convert  them  back  again  was  named  display.  We 
have  changed  these  names  to  imgprb  and  prbimg,  respectively.  The  main 
programs  now  invoke  subroutines  to  do  most  of  the  work;  we  have  called 
these  imgprbsub  and  prbimg  sub. 


*  Conversion  to  Testb  ed  Formats 

The  original  imgprb  and  prbimg  routines  accepted  images  no  wider  than  512 
pixels.  We  have  removed  this  restriction.  The  pixels  were  also  limited  to  B 
bits,  or  256  gray  levels.  We  have  extended  this  range  to  the  36-bit  pixels 
currently  handled  by  the  Testbed  image  access  software.  Testbed  images  of 
unusual  pixel  lengths,  e.g.,  3  bits,  are  supported  directly,  as  opposed  to  the 
UM  practice  of  padding  them  into  B-bit  fields  with  a  "significant  bits  per 
pixel"  specification  to  recover  the  dynamic  range  information 


*  Generalization  to  Multiple  Labels 

The  original  version  of  imgprb  assumed  that  only  two  labels  were  to  be  used, 
although  the  rest  of  the  package  did  accept  more  than  two  labels.  We  have 
extended  the  mapping  algorithm  as  described  in  the  previous  section.  The 
mapping  to  multiple  labels  was  chosen  to  be  the  inverse  of  the  mapping  back 
from  multiple  labels  that  was  already  implemented  in  prbimg . 

Prbimg  accepted  a  "number  of  labels"  argument,  but  then  ignored  it,  since 
this  information  could  be  more  reliably  obtained  from  the  header  of  the  pro¬ 
bability  file.  The  demonstration  shell  script  supplied  with  the  original  pack¬ 
age  erroneously  omitted  this  argument  in  calls  to  the  routine.  This  has  been 
fixed  by  eliminating  the  interactive  argument. 
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*  Default  e  d  Arguments 

The  main  programs  are  able  to  count  the  number  of  arguments  passed  to 
them  and  to  substitute  defaults  for  any  missing  arguments.  We  have 
retained  this  capability  in  the  RELAX  driver  program.  The  invoked  subrou¬ 
tines,  however,  must  be  supplied  with  a  full  set  of  arguments.  We  have 
adopted  the  convention  that  a  negative  minimum  or  maximum  range 
specification  passed  to  imgprbsub  or  prbimgsub  will  denote  that  the  full 
dynamic  range  of  the  picture  file  is  to  be  used.  The  user  should  specify  non¬ 
negative  values  if  stretching  and  clipping  are  desired. 


*  Input  dipping 

The  imgprb  routine  originally  stretched  imagery  to  the  specified  gray-level 
range,  but  did  not  clip  to  this  range.  The  mapping  to  a  probability  image  was 
fine  if  the  true  image  range  was  specified,  but  would  generate  probabilities 
that  were  negative  or  greater  than  unity  for  gray  levels  outside  this  range. 
The  result  of  the  inverse  prbimg  mapping  on  such  a  file  was  a  sawtooth  func¬ 
tion.  We  have  corrected  this  by  clipping  all  probability  values  to  the  0. 0-1.0 
range. 


*  Word  Size  Conversion 

The  original  package  was  written  for  a  PDP-11  computer,  for  which  the  C 
language  uses  16-bit  integers.  Our  installation  is  on  a  VAX  11/780  that  uses 
32-bit  integers.  This  difference  is  important  in  the  writing  and  reading  of 
neighborhood  and  compatibility  file  headers,  since  the  I/O  statements 
specified  the  number  of  bytes  to  be  transferred.  We  have  rewritten  these 
sections  to  use  the  C  sizeofQ  construct,  which  is  guaranteed  to  be  valid  on 
any  type  of  machine.  The  relaxation  data  files,  however,  cannot  be 
transferred  between  machines  with  different  integer  sizes  unless  further 
standardization  is  implemented. 

The  need  for  additional  changes  became  apparent  during  our  software  evaluation 
effort.  Many  of  the  needed  improvements  have  to  do  with  the  command  interpreter  or 
the  package  philosophy  rather  than  the  relaxation  algorithms.  We  suggest  the  follow¬ 
ing  implementation  changes: 


*  Initialization  Capability 

The  program  should  be  able  to  run  a  startup  file.  This  would  allow  the  user 
to  customize  the  package  to  his  own  preferences  and  tasks.  Additional  flexi¬ 
bility  should  be  built  into  the  command  driver  so  that  it  could  take  advan¬ 
tage  of  initial  profile  information  to  set  default  neighborhood  sizes  and  file 
names. 
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*  Re  definition  of  Pro  b  abilities 

The  imgprb  two-label  mapping  should  be  reversed  to  match  its  multilabel 
mapping  interpretation..  The  prbimg  inverse  mapping  should  offer  an  option 
as  to  whether  two-label  probability  files  will  be  mapped  to  gray  Levels  or  to  a 
binary  output.  Possibly  a  separate  routine  should  be  provided  for  each  map¬ 
ping  and  inverse  mapping  function  instead  of  combining  many  functions  in 
one  routine. 


*  Defcom  Improvement 

The  defcom  routine  for  interactively  defining  compatibility  coefficients  is 
very  tedious  to  use.  It  is  often  easier  to  construct  a  file  containing  the 
coefficients  and  then  pipe  it  into  defcom ,  using  the  command  interpreter's 
"<”  facility  for  acquiring  command  input  directly  from  a  text  file.  If 
coefficients  are  provided  in  this  manner  and  a  neighborhood  file  is  not 
specified,  an  inconsistency  arises:  the  neighborhood  size  must  be  included  at 
the  start  of  the  piped  coefficients,  even  though  they  would  not  be  needed  if 
the  coefficients  were  entered  interactively.  This  should  be  changed  so  that 
the  routine  reading  the  coefficients  does  not  expect  to  read  the  neighbor¬ 
hood  size  as  well. 


*  Rim-Time  Argument  Passing 

The  command  interpreter’s  "<"  mechanism  for  invoking  script  files  should 
accept  arguments.  The  UNIX  shell  languages  provide  a  good  model  for  the 
type  of  argument  macro  expansion  capability  that  is  required. 


*  Automatic  Checkpointing 

If  automated  iteration  is  ever  added  to  the  RELAX  package,  there  should  be 
some  easy  method  of  saving  a  checkpoint  output  every  few  iterations.  Relax¬ 
ation  is  such  an  expensive  technique  that  a  user  should  not  have  to  start 
again  from  scratch  if  the  system  crashes  or  if  processing  has  gone  past  the 
optimum  point  and  begun  to  degrade  the  image. 


*  Pile  Name  Flexibility 

Part  of  the  checkpointing  problem  is  due  to  the  current  "hard-wiring”  of  the 
names  prb. img  and  compat.dat  into  several  of  the  routines.  This  causes  the 
hrelax  and  pretax  routines  to  overwrite  the  prb.img  file,  making  it  difficult  to 
repeat  an  iteration  (e.flr.,  with  different  parameters)  or  to  recover  after  too 
many  iterations.  It  is  also  difficult  to  remember  exactly  which  iteration  or 
processing  sequence  generated  the  current  prb.img  file.  Although  the 
UNIX/EUNICE  hierarchical  file  system  and  the  EUNICE  multiple  file-version 
facility  alleviate  some  of  these  problems,  the  best  solution  is  to  allow  arbi¬ 
trary  file  names  to  be  passed  to  the  processing  routines.  An  intelligent  sys¬ 
tem  for  constructing  default  file  names  could  also  be  helpful. 
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*  Subroutine  Implementation 

The  RELAX  routines  are  currently  implemented  as  stand-alone  main  pro¬ 
grams  that  can  also  be  invoked  from  the  RELAX  driver.  This  differs  from  the 
subroutine-based  implementation  that  is  common  to  all  other  Testbed 
software.  Although  the  main-program  approach  works,  it  is  difficult  to 
integrate  with  the  rest  of  the  Testbed.  Our  directory  hierarchies  and  archiv¬ 
ing  techniques  are  based  on  libraries  of  subroutines  rather  than  on  clusters 
of  main  programs.  It  would  be  simpler  to  maintain  a  system  that  has  a  uni¬ 
form  programming  philosophy. 

As  an  example,  consider  the  compilation  of  the  fcrelaz  routine.  Ordinarily, 
each  main  program  in  the  Testbed  is  accompanied  by  a  makefile  script  that 
“remembers”  all  the  include  files  and  libraries  needed  in  compiling  the  rou¬ 
tine.  If  hrelax  is  to  be  created  by  the  setup.csh  script  provided  by  UM,  the 
compilation  command  must  be  in  that  script.  If  it  is  to  be  created  by  the 
RELAX  program,  the  compilation  command  must  be  compiled  into  that  code. 
Thus,  changes  in  the  structure  or  location  of  the  include  files  and  libraries 
must  now  be  implemented  by  changes  in  several  types  of  source  code.  'While 
this  is  not  difficult,  it  is  certainly  more  trouble  than  updating  a  single  type  of 
file  used  consistently  throughout  the  system. 

The  chief  reason  for  using  main  programs  rather  than  subroutines  is  that 
optimized  relaxation  operators  are  compiled  for  each  specific  task.  This 
exchanges  extra  compilation  time  for  reduced  execution  time— which  seems 
reasonable,  given  the  length  of  time  currently  required  to  run  a  relaxation 
step/  Separately  compiled  subroutine  modules  could  be  linked  into  the 
driver  package  at  run  time,  although  this  UNIX  capability  is  not  easily  acces¬ 
sible  or  commonly  used. 

The  shell  languages  do  provide  convenient  mechanisms  for  sequencing  pro¬ 
grams,  but  they  are  better  suited  for  batch  execution  than  for  interactive 
use.  Furthermore,  they  are  general-purpose  and  hence  lack  the  focused 
environment,  vocabulary,  defaults,  and  help  facilities  that  a  dedicated  com¬ 
mand  driver  can  offer.  Command  interpreters  can  invoke  either  main  pro¬ 
grams  or  subroutines,  but  are  somewhat  easier  to  write  for  the  subroutine 
case. 

Another  benefit  of  subroutine-based  implementation  is  control  of  interpro¬ 
cess  communication.  Programs  invoked  by  shell  scripts  can  communicate 
only  via  files.  (Communication  by  means  of  shell  environment  variables  is 
also  possible,  but  not  commonly  done.)  File  passing  is  rather  awkward,  since 
it  requires  each  main  program  to  open,  read,  and  parse  every  file.  It  also 
tends  to  clutter  the  environment  with  superfluous  files;  these  can  be  deleted 
by  commands  at  the  end  of  a  shell  script,  but  must  be  removed  by  hand  if 
the  session  is  interactive  oris  aborted 

A  more  traditional  subroutine-based  programming  style  would  allow  com¬ 
munication  by  means  of  global  variables  and  passed  arguments  as  well  as 
files.  Display  devices  would  also  be  under  better  control,  since  the  device 
status  can  be  remembered  and  maintained  by  the  driver  program,  and  each 
routine  need  not  reallocate  or  reinitialize  the  display  to  be  sure  of  getting  a 
usable  configuration. 

In  addition,  the  data  files  could  be  opened  once,  passed  around,  then  closed 
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or  deleted.  Permanent  files  need  be  created  only  for  permanent  data,  and 
the  user  need  not  specify  the  file  names  as  arguments  to  every  routine. 

A  final  advantage  of  the  subroutine  approach  is  that  there  is  less  system 
overhead.  The  current  approach  requires  that  a  UNIX  (or  EUNICE)  process 
be  created  to  run  each  command,  since  this  is  the  only  way  to  invoke  amain 
program.  The  overhead  in  creating  a  process  is  much  greater  than  that  of 
calling  a  subroutine,  and,  in  fact,  may  require  several  seconds  of  real  time. 


*  Operator  Libraries 

As  mentioned  above,  one  advantage  of  the  University  of  Maryland  approach  is 
the  run-time  compilation  of  customized  image  operators  in  order  to  reduce 
total  execution  time.  The  actual  speedup  achieved  in  the  current  RELAX 
package  is  rather  small,  but  the  technique  could  be  extended  for  larger 
gains.  Perhaps  the  greatest  speedup  could  be  achieved  by  using  replicated 
in-line  code  instead  of  iterative  constructs.  For  research  purposes,  of 
course,  such  optimizations  are  seldom  worthwhile. 

The  run-time  compilation  of  such  routines  is  not  a  problem.  It  can  be  done 
as  a  separate  program  step,  possibly  invoked  interactively  by  interrupting  a 
session  (with  compiling,  and  then  resuming.  It  can  also  be  done  by 
using  the  "system"  subroutine  to  call  the  compiler  from  within  another  pro¬ 
gram. 

Since  compilation  is  an  expensive  step,  it  might  make  sense  to  keep  the 
most  useful  operators  in  a  library  of  executable  files.  Nothing  in  the  current 
RELAX  package  prevents  this  from  being  done,  but  neither  are  there  any 
features  to  facilitate  it.  At  the  very  least,  a  naming  convention  should  be 
devised  so  that  the  operator  names  can  be  remembered. 


*  Speedup 

The  current  relaxation  updating  algorithms  are  exceedingly  slow.  (They  take 
about  three  CPU  minutes  for  one  pass  of  a  3x3  operator  through  a 
13B  x  128  image.)  This  is  probably  due  to  an  inefficient  scheme  for  accessing 
the  floating-point  label  probabilities.  Further  investigation  is  needed  to 
determine  the  requisite  time  for  this  type  of  processing. 
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This  section  constitutes  a  user's  guide  to  the  RELAX  package  as  it  is  implemented  on 
the  SRI  Image  Understanding  Testbed.  As  with  any  reference  manual,  it  has  occasion¬ 
ally  been  necessary  to  refer  to  terms  before  they  are  defined  and  discussed  in  detail. 
The  first-time  reader  may  find  a  preliminary  scan  through  the  section  helpful.  Addi¬ 
tional  information  is  available  online,  as  described  below. 


5.1.  Interactive  Usage 

There  are  typically  five  steps  in  applying  relaxation  to  an  image: 

*  Compilation  of  the  updating  routine 

*  Image-to-probability  mapping 

*  Estimation  of  compatibility  coefficients 

*  Relaxation  updating  iterations 

*  Probability-to-image  mapping, 

Display  steps  are  usually  interspersed  so  that  one  can  watch  the  progress  of  the 
enhancement.  Other  techniques  are  sometimes  required,  such  as  edge  detection  or 
manual  entry  of  neighborhood  and  compatibility  data.  All  of  these  processes  can  be 
invoked  from  within  the  RELAX  package. 

The  RELAX  package  currently  takes  no  command-line  arguments;  just  type  "relax” 
and  begin  an  interactive  session.  The  following  sample  session  displays  some  of  the 
capabilities  of  the  underlying  Cl  driver  language. 


re  1  si 

RELAX,  Version  1.0 


> 

This  invokes  the  program.  You  need  not  specify  the  full  directory  path  name  for  the 
executable  file  if  the  path  is  given  in  your  UNIX  .cshrc  shell  startup  file.  (If  you  have 
no  startup  file,  you  may  have  to  specify  /Lu/tb  /bin /-relax  or  some  other  full  path 
name.)  The  system  then  responds  and  waits  for  commands. 

>  • 


dcfc om 
defnbr 
erase 
h  c  omp  a  t 
hr e  1  ax 
imgprb 
i nser  t 


pcompat 

prbimg 

pretax 

quit 

setup 

he  1  p 
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An  command  lists  all  available  commands.  The  help  command  is  provided  by  the 
Cl  driver;  all  others  are  specifically  related  to  the  relaxation  system.  Typing  "help" 
•will  give  further  information  on  the  Cl  command  interpreter  and  the  help  subsystem. 

>  he  1  p 

Command  names,  variable  names,  and  help  topics 
may  he  abbreviated  to  any  unique  prefix.  Several 
instructions  may  appear  on  the  same  line,  separated 
by  semicolons.  Use  ~ O  to  abort  typeout  and  ** Z 
to  exi t . 

c  oilman  d  [  arg  ...  ] 

Execute  a  command  with  the  specified  arguments. 
loo* 

List  commands  that  begin  vith  "Too". 

Use  just  to  list  all  commands. 

f  oo*  = 

List  the  names  and  values  of  variables 
that  begin  vith  "Too" . 

variable  [  param  ...  ] 

Display  a  variable  value.  Some  variables 
may  require  subscripts  or  parameters. 

variable  [  param  ...  ]  =  value 

Assign  a  variable  value.  The  equals  sign  ( ~ ) 
may  appear  anywhere. 

?  topic  <or>  help  topic  <or>  help  * 

Print  help  message  related  to  topic.  If  topic 
is  ambiguous,  list  the  names  of  matching  topics. 

pu sh_l  eve  1 

Creates  a  new  level  of  the  Cl  driver  vith  the  same 
commands  available  as  there  were  at  the  preceding 
level.  The  user  may  execute  any  combination  of 
commands  before  quitting. 

1  conmand 

Fork  a  shell  and  execute  the  UNIX  conmand.  If 
no  conmand  is  given,  just  create  an  interactive 
she  11  process. 

<  conmand f  i  1  e 

Read  commands  from  " conmand f i 1 e" . 

;  conxnent 

Accept  a  consent  line;  take  no  action. 

As  an  example  of  the  online  help  facility,  we  can  print  out  the  contents  of  the 
setup.txt  file  in  the  relaxation  system  help  directory. 
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>  help  setup 

letup  {hjpj  n labels  [ncols  iron] 

Setup  is  med  to  create  programs  for  relaxation  operations 
on  images.  Both  a  compatibility  operator  and  a  relaxation 
operator  will  be  compiled.  Either  Hnmne  1 -Zicker -Rosenf e 1 d 
(h)  or  Peleg  (p)  relaxation  formulas  may  be  chosen;  the 
corresponding  programs  vi 1 1  be  hcompat  and  hrelax  or  pcompat 
and  prelax.  The  default  is  "h".  Ton  may  also  specify  the 
number  of  class  labels  (default  2)  to  be  used  and  the  sixe 
of  the  relaxation  neighborhood  (default  3x3). 

Running  the  setup  command  with  default  arguments  produces  a  param  file  that  is 
compiled  into  the  hrelax  and  hcompat  programs.  The  param  file  is  then  deleted  and 
the  executable  programs  are  left  in  the  current  directory. 

>  setup 

Creating  the  Humne  1 -Zucker -Bo senf e 1 d  relaxation  package  ... 

Creating  the  "param”  file  ... 

Compiling  hcompat  ... 

Compiling  hrelax  ... 

Removing  the  param  file  ... 

Setup  completed. 

‘  A  UNIX  directory  listing  command  shows  the  new  executable  programs  and  a  demon¬ 
stration  command  file  that  was  created  previously. 

>  ill 

hcompat.exe  hrelax.exe  tank.cmd 

[ cont inu i ng ] 

Here  we  use  the  command  file  to  run  a  relaxation  sequence.  If  the  file  were  not  in 
this  directory,  RELAX  would  search  for  it  in  directory  /iu/tb/src  /relax /demo . 
RELAX  then  echos  the  commands  and  performs  the  indicated  actions  just  as  if  they 
had  been  typed  from  the  terminal. 

>  < t ank .  cmd 


The  image  is  first  converted  to  two-label  probability  form.  It  has  no  pixel  values 
below  13  or  above  49,  so  stretching  and  clipping  are  requested.  (This  speeds  conver¬ 
gence  and  improves  the  appearance  of  the  output  image.) 

>  :  Convert  the  image  to  probability  form. 

>  imgprb  /iu/tb/pi c/tank/bv. img  2  13  49 

Next  the  compatibility  coefficients  are  needed.  They  are  computed  from  the  pixel 
relationships  in  the  original  image.  They  could  also  be  entered  by  hand  with  the 
defnbr  and  defcom  commands.  The  program  echos  the  hcompat  request,  with  the 
default  file  names  filled  in. 

>  :  Compute  the  compatibility  coefficients. 

>  hcompat 

hcompat  prb.img  compat.dat 


\ 
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The  original  image  is  now  to  be  reconstructed  and  displayed  in  an  upper-left  area  of 
the  screen.  This  reconstruction,  which  shows  the  input  after  stretching  and  clipping, 
also  serves  as  a  check  on  the  imgprb  and  prbimg  processes. 

>  :  Display  the  (stretched)  original  image. 

>  erase 

>  prbimg  ontputO. img 

>  insert  outpntO. img  100  348 

Now  eight  relaxation  steps  are  to  be  run.  Each  will  produce  an  output  image;  these 
images  will  be  displayed  along  with  the  original  in  a  3  x  3  pattern. 

>  :  Display  eight  relaxation  steps. 

>  hrelax 

hrelax  prb.img  compat.dat 

>  prbimg  ontputl.img 

>  insert  ontputl.img  224  348 

>  hrelax 

hrelax  prb.img  compat.dat 

>  prbimg  outputs,  img 

>  insert  output2.img  348  348 

>  hrelax 

hrelax  prb.img  compat.dat 

>  prbimg  output3. img 

>  insert  output3. img  100  Z24 

>  hrelax 

hrelax  prb.img  compat.dat 

>  prbimg  output4.  img 

>  insert  output4. img  ZZ4  2Z4 

>  hrelax 

hrelax  prb.img  compat.dat 

>  prbimg  output5.  img 

>  insert  output5. img  348  224 

>  hrelax 

hrelax  prb.img  compat.dat 

>  prbimg  outputs,  img 

>  insert  outputs. img  100  100 

>  hrelax 

hrelax  prb.img  compat.dat 

>  prbimg  output7.  img 

>  insert  output7. img  224  100 

>  hrelax 

hrelax  prb.img  compat.dat 

>  prbimg  outputs. img 

>  insert  outputs. img  348  100 

End  of  c oilman d  file. 


The  quit  command  is  now  given  to  return  the  user  to  the  operating-system  level. 


23 


Program  Documentation 


>  qui  t 


5.2.  Commands 

The  following  commands  are  currently  available  within  the  RELAX  system: 

defcom  compatfile  relaxtype  nlabels  [neighborfile] 

Defcom  is  an  interactive  program  used  to  install  manually  computed  com¬ 
patibility  coefficients  for  each  neighbor  of  a  point.  The  arguments  specify 
the  file  to  which  the  coefficients  are  to  be  written,  whether  a  Hummel- 
Zucker-Rosenfeld  ("h")  or  Peleg  ("p")  relaxation  is  desired,  the  number  of 
labels  in  the  relaxation  process,  and,  optionally,  a  file  that  specifies  a  non¬ 
standard  neighborhood. 

defnbr  neighborfile  ncola  nrowa 

Defnbr  is  an  interactive  program  used  to  define  a  nonstandard  neighbor¬ 
hood  for  each  point.  The  "neighborfile"  argument  specifies  the  file  to 
which  the  neighborhood  definition  is  to  be  written.  The  number  of  columns 
and  rows  that  contain  the  neighborhood  must  also  be  specified.  The  pro¬ 
gram  will  ask  which  is  to  be  considered  the  center  point  and  whether  each 
neighbor  is  to  be  considered  as  part  of  the  neighborhood. 

erase 

Erase  the  display.  Note  that  the  display  is  not  allocated  or  locked,  so  it  will 
remain  clear  only  if  no  one  else  transmits  data  to  it. 

hcompat  prbfile  compatfile  [neighborfile] 

Computes  the  Hummel-Zucker-Rosenfeld  compatibility  coefficients  for  the 
probability  file  (default  prb.img )  and  stores  them  in  a  compatibility  file 
\compat.dat).  You  may  specify  a  neighborhood  file  as  created  by  defnbr. 

The  hcompat  program  must  have  been  compiled  previously:  see  setup.  If 
you  would  like  to  specify  the  compatibility  coefficients  by  hand,  see 
defcom. 

hrelax  prbfile  compatfile  [neighborfile] 

Performs  one  Hummel-Zucker-Rosenfeld  relaxation  operation  on  a  proba¬ 
bility  file  (default  prb.img ),  using  compatibility  coefficients  in  compatfile 
{compat.dat).  You  may  specify  a  neighborhood  file  as  created  by  defnbr. 

The  hrelax  program  must  have  been  compiled  previously:  see  setup. 

imgprb  ini  mg  nlabels  minval  maxval 

Convert  an  image  to  a  probability  format  with  the  specified  number  of 
labels.  The  image  will  be  clipped  (stretched)  using  minval  and  maxval  as 
the  outer  gray-level  limits:  omit  these  or  specify  -1  if  the  full  input  range  is 
to  be  used.  The  file  prb.img  will  be  produced  as  output.  At  present  it  is  a 
floating-point  data  file,  not  a  true  picture  file. 

insert  picname  mincol  m inrow 

Insert  the  picture  into  the  display  at  the  specified  lower-left  pixel  position. 
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pcompat  prbfile  compatfile  [neighborfile] 

Computes  the  Peleg  compatibility  coefficients  for  the  probability  file 
(default  prb.img)  and  stores  them  in  a  compatibility  file  (campat.dat).  You 
may  specify  a  neighborhood  file  as  created  by  defribr. 

The  pcompat  program  must  have  been  compiled  previously:  see  setup.  If 
you  would  like  to  specify  the  compatibility  coefficients  by  hand,  see 
defcom. 

prbimg  outname  minval  maxval 

Convert  a  probability  file  to  a  luminance  image  format  with  the  specified 
output  range.  Omit  minval  and  maxval,  or  specify  -1  to  use  the  full  (8-bit) 
dynamic  range  of  the  output  image.  The  input  file  is  assumed  to  be 
prb.img. 

prelax  prbfile  compatfile  [n  neighborfile ]  [s  nsets  sizel  ...] 

Performs  one  Peleg  relaxation  operation  on  a  probability  file  (default 
prb.img),  using  compatibility  coefficients  in  compatfile  (c07n.pt1i.daf).  You 
may  specify  a  neighborhood  file  as  created  by  defnbr;  precede  it  with  the 
letter  n. 

You  may  also  specify  the  grouping  of  label  values  into  sets.  Previously,  in 
Section  3.2,  we  stated  the  Peleg  updating  rule  as 


fc  =  1,  •  •  1  ,71 


where  the  denominator  is  the  sum  over  the  set  of  all  labels.  Actually,  this 
sum  can  be  limited  to  a  set  of  labels,  rather  than  summing  over  all  labels. 
The  sum  is  carried  out  over  the  set  to  which  the  label  A*.  belongs. 


*#«)(»,)  =  -1_E 

m  '  E  jx"  W’M 


fc  =  l.  •  ■  •  ,71 


Label  sets  are  specified  by  the  command  option  [s  nsets  sizel  ...],  where 
nsets  specifies  the  number  of  different  sets  and  sizel,  size2,  ....  the  number 
of  labels  in  each  set.  The  sets  must  consists  of  consecutive  labels,  e.g.,  [s  3 
2  4  3]  specifies  that  there  are  3  sets  of  labels;  the  first  set  contains  the  2 
labels  0,  and  1;  the  second  set  labels  2,  3,  4,  and  5;  and  the  third  set  labels 
6,  7,  and  8. 


Although  the  added  flexibility  of  this  label  set  grouping  is  available,  its  use 
is  not  recommended.  The  numbers  computed  by  this  method  are  no 
longer  probabilities  and  the  behavior  of  the  process  cannot  be  predicted1. 

The  prelax  program  must  have  been  compiled  previously:  see  setup. 


fAzriei  Rosenfeld,  personal  communication,  1983. 
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quit  n 

Quit  the  current  level  of  the  Cl  driver.  At  the  top  level  this  will  leave 
RELAX.  The  argument  n  may  be  provided  to  quit  more  than  one  level. 
Specify  -1  or  some  large  number  to  abort  all  levels  of  the  driver  and  exit 
from  the  program. 

setup  (h|p)  nlabels  [ncols  nrows] 

Setup  is  used  to  create  programs  for  relaxation  operations  on  images. 
Both  a  compatibility  operator  and  a  relaxation  operator  will  be  compiled. 
Either  Hummel-Zucker-Rosenfeld  (h)  or  Peleg  (p)  relaxation  formulas  may 
be  chosen;  the  corresponding  programs  will  be  hcompat  and  hrelax  or 
pcompat  and  pretax.  The  default  is  "h”.  You  may  also  specify  the  number 
of  class  labels  (default  2)  to  be  used  and  the  size  of  the  relaxation  neigh¬ 
borhood  (default  3  x  3). 


5.3.  Batch  Execution 

The  RELAX  program  offers  two  methods  of  invoking  prestored  commands.  The  first  is 
the  interactive  invocation  of  Cl  command  files,  as  illustrated  earlier.  The  second  is  to 
drive  the  entire  RELAX  session  from  an  operating-system  script.  A  UNIX  shell  script 
invoked  by.  e.g.,  "runrelax  picture.img  10  195",  might  look  like  the  following: 

#  This  ia  file  "nmrelai". 

f 

#  Argument  Si  ia  .the  gray-level  image. 

#  Argument  a  S2  and  S3  are  the  optional  low  and 
f  high  clipping  values. 

relax  << I 

:  Hake  3x3  neighborhood,  two  label, 

:  HZR  coefficient  computation  and 
:  relaxation  programs, 
setup  h  2 

:  Display  the  original  image.  Si. 
erase 

insert  Si  100  348 

:  Transform  the  picture  into  a  probabilistic  image, 
imgprb  Si  prb.img  2  S2  S3 

:  Compute  the  compatibility  coefficients, 
hcompat  prb.img  compat.dat 

:  Apply  the  relaxation  operator, 
hrelax  prb.img  compat.dat 

:  Display  the  smoothed  image, 
prbimg  prb.img  outputl.img 
insert  outputl.img  224  34B 


echo  "Finished." 

Note  that  the  shell  substitution  mechanism  can  be  used.  This  is  an  advantage  over 
interactive  use,  in  which  no  substitution  of  variables  is  currently  implemented. 
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To  save  the  typed  terminal  output,  you  should  pipe  the  standard  output  into  a  file. 
The  UNIX  method  for  doing  this  is  to  add  >session.log  to  the  relax  command  within 
the  script  or  to  the  UNIX  command  line  that  invokes  the  script.  You  may  also  use  the 
UNIX  script  or  tee  commands  to  route  the  typed  output  simultaneously  to  a  file  and 
to  your  terminal.  (UNIX  output  buffering  generally  prevents  you  from  interacting 
•with  a  program  while  the  script  is  being  created.  The  EUNICE  operating  system  does 
not  have  this  limitation.) 

The  actual  submission  of  this  shell  script  is  described  in  the  UNIX  programmer's 
manual.  You  should  run  it  in  foreground  mode  if  you  want  to  interact  with  the  pro¬ 
gram.  If  you  run  it  in  background  mode,  be  sure  to  pipe  the  output  into  a  log  file  so 
that  it  won’t  appear  on  your  terminal.  You  can  monitor  the  log  file  during  execution, 
using  the  caf  or  tail  -/  [formerly  fra]  commands  to  make  sure  everything  is  running 
smoothly,  although  the  UNIX  buffering  mechanism  prevents  you  from  monitoring  in 
reeil  time.  You  can  also  halt  the  process  or  reconnect  it  to  your  terminal  if  you  wish. 
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This  section  documents  the  performance  of  the  Hummel-Zucker-Rosenfeld  and  Peleg 
relaxation  algorithms  on  a  variety  of  imagery  and  in  several  scene  analysis  tasks. 
Although  we  have  chosen  realistic  tasks,  our  goal  was  not  the  full  exploration  of  relaxa¬ 
tion  techniques  for  these  applications.  Rather,  we  have  chosen  tasks  with  simple  map¬ 
ping  functions  to  and  from  the  probability  domain  so  that  we  could  better  assess  the 
actions  of  the  probability-updating  functions. 

We  could  devise  objective  performance  measures  for  particular  tasks  [FeketeBl],  but 
the  relationship  of  such  measures  to  subjective  scene  analysis  performance  would  be 
difficult  to  quantify.  Linear-feature  extraction,  for  instance,  must  be  rated  differently 
when  we  desire  only  prominent  features  (e.gr.,  for  stereo  image  matching)  or  closed 
boundaries  of  regions  than  when  we  desire  extraction  of  all  detectable  discontinuities. 

.We  could  also  compute  performance  measures  for  restoration  of  images  to  which  we 
have  added  blur  or  noise,  but  we  would  need  models  of  the  imagery  and  degradations 
occurring  in  realistic  scenarios.  If  such  models  were  available,  other  methods  of  image 
restoration  would  almost  certainly  be  more  effective  than  heuristic  relaxation.  Furth¬ 
ermore,  any  comparison  of  relaxation  output  with  ground  truth  would  necessarily 
depend  on  the  function  used  for  mapping  the  initial  image  to  the  probability  domain. 

We  have  therefore  chosen  a  subjective  evaluation  procedure,  We  compare  the  relaxa¬ 
tion  output  with  the  probability  input  after  each  has  been  mapped  back  to  the  lumi¬ 
nance  domain.  This  type  of  comparison  is  valid  for  almost  any  task.  It  measures 
whether  the  relaxation  operation  has  made  the  image  more  useful  for  the  intended 
application. 

The  particular  scene  analysis  tasks  we  have  selected  are  discussed  below.  For  any  task 
and  type  of  imagery,  we  still  must  choose  the 

*  Image-to-probability  and  inverse  mappings 

*  Size  (and  shape)  of  the  compatibility  neighborhood 

*  Method  of  computing  compatibility  coefficients 

*  Relaxation  scheme  (HZR  or  Peleg) 

*  Number  of  relaxation  iterations. 

We  shall  explore  these  choices  in  the  following  subsections  and  then  summarize  our 
conclusions. 


6.1.  Task  Selection 

The  RELAX  package  supplied  by  the  University  of  Maryland  contained  a  demonstra¬ 
tion  of  noise  suppression  on  an  infrared  image  of  a  military  tank.  The  nature  of  the 
image,  a  single  bright  tank  centered  against  a  dark  background,  made  this  also  a 
demonstration  of  object  detection  and  of  segmentation  in  a  noisy  image.  We  have 
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used  this  image  (and  the  implied  tasks)  extensively  in  learning  to  run  the  RELAX 
package  and  to  understand  its  functioning.  Two  other  test  tasks  we  have  selected  are 
edge  linking  and  anomaly  detection. 

Edge  linking  is  the  enhancement  of  linear  features  in  an  image  by  strengthening  con¬ 
nected  edge  elements  and  suppressing  isolated  or  conflicting  ones.  A  set  of  edge 
operators  is  first  passed  over  the  image;  they  return,  for  each  pixel,  the  probabilities 
that  scene  edges  at  various  orientations  are  present.  (We  shall  treat  no-edge  as  a 
label  also.)  The  relaxation  stage  strengthens  the  probabilities  of  edge  labels  that  link 
"nose  to  tail"  with  the  edges  at  neighboring  pixels  while  suppressing  other  edge  pair 
orientations. 

Anomaly  detection  is  the  identification  or  enhancement  of  isolated  blobs  against  a 
fairly  uniform  background,  An  image  operator  first  classifies  pixels  as  either  back¬ 
ground  or  anomalous  according  to  the  statistics  of  the  background  and  the  gray  level 
of  each  pixel.  Relaxation  then  reinforces  the  classification  of  a  pixel  if  its  neighbor¬ 
ing  pixels  are  similarly  classified.  This  two-label  operation  may  also  be  viewed  as  a 
noise-suppression  application. 


6.2.  Edge  linking 

The  first  step  in  edge  linking  is  to  calculate  the  initial  probabilities  that  scene  edges 
*  at  various  orientations  pass  through  each  pixel. '  We  convolve  the  image  with  four 
Sobel-type  masks  that  detect  northwest ,  north,  northeast,  and  east  edges.  Because 
we  consider  opposing  gradient  directions  (or  contrasts)  to  be  equivalent,  there  are 
four  orientations  rather  than  eight. 

The  3x3  convolution  operators  are: 
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Five  labels  are  then  attached  to  each  pixel:  no-edge,  northwest ,  north,  northeast, 
and  east.  The  probabilities  for  each  of  these  labels  are  calculated  from  the  outputs 
of  the  four  convolution  operators.  Op  (dir).  For  label  north,  the  probability  at  a  par¬ 
ticular  pixel  is  calculated  as 


Op  ( north ) 

Prob  (north)  -  - 

£  Op  {dir) 
all  c fir 


max  Op  {dir) 

aUdtr 


max  Op  (dir) 

all  (Hr, 


tdlimagt 

pixwu 


and  similarly  for  Prob  {northwest),  Prob  ( northeast ),  and  Prob  {east).  The  first  term 
relates  the  Op  {north)  response  to  the  response  of  the  other  edge  operators;  the 
second  compares  the  operator  response  at  that  pixel  with  responses  over  the  entire 
image, 

The  probability  of  no-edge  is  calculated  as 


29 


Evaluation 


Prab  (710-edge)  =  1  — 


max  Op  (dir) 

all  dir 


max  Op(dir) 

all  imapw 


Together  these  probabilities  constitute  the  initial  probability  image, 
schemes  have  been  reported  in  the  literature  [Riseman77.  Zucker77]. 


Similar 


The  next  step  is  to  calculate  the  compatibility  coefficients.  We  have  generally  used 
the  RELAX  package  hcampat  and  pcampat  routines  for  that  purpose.  Our  aim  was  to 
evaluate  fully  automated  relaxation  rather  than  to  develop  optimal  compatibility 
matrices  for  this  one  application;  the  former  is  the  usual  approach  in  the  relaxation 
literature. 


Figures  1-13  illustrate  the  results  of  our  edge-linking  efforts.  Each  figure  consists  of 
sixteen  pictures.  The  upper-left  picture  is  the  original  image.  The  other  three  pic¬ 
tures  across  the  top  of  the  figure  and  the  leftmost  picture  on  the  second  row  are  the 
northwest ,  north,  northeast ,  and  easf  edge  maps.  In  these  images,  black  represents 
zero  probability  of  an  edge,  white  a  unity  probability. 

The  remaining  pictures  are  binary  Images.  The  second  from  the  left  in  the  second 
row  is  the  inverse  mapping  of  the  original  probability  image.  It  is  produced  from  the 
four  edge  maps  by  displaying,  for  each  pixel,  the  label  with  the  highest  probability; 
no-edge  maps  to  black  while  all  other  labels  map  to  white.  The  remaining  ten  pic¬ 
tures  are  obtained  by  successive  iterations  of  the  relaxation  procedure.  They 
represent  the  results  of  1,  2,  ....  10  iterations  of  relaxation.  Figure  2  is  the  exception: 
the  ten  pictures  are  the  results  of  10,  20 . '100  iterations. 

Figures  1  and  2  illustrate  a  fairly  typical  relaxation  application.  The  first  few  itera¬ 
tions  of  relaxation  show  a  strong  edge-linking  effect.  Later  iterations  seem  to  do  lit¬ 
tle  except  smooth  or  blur  the  enhanced  structures.  This  dual  nature  of  relaxation 
has  been  analyzed  by  Richards  [RichardsBO].  Figure  3  shows  that  the  Peleg  relaxa¬ 
tion  scheme  exhibits  essentially  the  same  behavior. 

Figure  4  illustrates  the  HZR  method  on  aerial  imagery.  The  improvement  in  the 
displayed  output  is  rather  dramatic,  although  it  may  be  partially  due  to  the  method 
of  output  mapping. 

Figures  5  and  6  illustrate  the  effect  of  larger  neighborhoods.  The  7x7  neighborhood 
increases  edge  spreading  for  the  HZR  method  and  decreases  it  for  the  Peleg  method. 
Computation  time  is  much  greater  for  large  neighborhoods,  of  course. 

Figure  7  shows  that  strong  edge  structure  does  not  guarantee  effective  compatibility 
coefficients  if  the  edges  appear  equally  in  many  orientations  and  relationships.  This 
is  opposite  to  the  effect  of  most  enhancement  operators. 

Figure  8  shows  the  result  of  "enhancing"  the  edges  in  an  image  that  has  no  percep¬ 
tual  edge  structure.  The  procedure  for  computing  compatibility  coefficients  regis¬ 
ters  any.  regularities  in  the..imageT  .not  just  strong  .responses  from  the  edge  detectors. 
This,  combined  with  the  relaxation  blurring  effect,  gives  a  surprisingly  good  noise- 
suppression  or  object  detection  operator. 

Figures  9  through  12  show  that  the  exact  values  of  the  compatibility  coefficients  are 
not  critical.  Figure  9  was  made  with  coefficients  rounded  to  one  decimal  place,  Fig¬ 
ure  10  with  these  same  numbers  doubled.  (Doubling  the  coefficients  will  have  no 
effect  on  Peleg  relaxation.)  Figure  11  was  generated  using  compatibility  coefficients 
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derived  from  the  image  in  Figure  12;  Figure  12  was  likewise  generated  using  compati¬ 
bility  coefficients  derived  from  the  image  in  Figure  11,  In  each  case  the  tampering 
had  relatively  little  effect. 

Figure  13  shows  what  happened  when  we  supplied  the  coefficients  by  hand  to  see  if  a 
careful  choice  of  the  values  could  produce  faster  relaxation  effects.  Although  these 
coefficients  were  intuitively  derived,  they  perform  very  badly.  For  a  pixel  and  the 
neighbor  above  it.  the  coefficients  we  used  for  the  Hummel-Zucker-Rosenfeld  scheme 
are 


Coe ff  ( north ,  north-west ) 
Coeff  (north ,  north) 

Cbeff  (north,  northeast ) 
Coeff  (north,  east) 

Coeff  (north,  no-edge) 

Coeff  (northeast ,  northwest) 
Coeff  (northeast ,  northeast) 
Coe f f  (northeast ,  east) 
Cbeff  (northeast ,  no-edge) 
Coeff  (eosf .  easf) 

Coeff  (ensf ,  no-edge) 

For  a  pixel  and  its  northeast  neighbor  we  used 

Coeff  (north ,  north-west ) 

Coe f  f  (north,  north) 

Coeff  (north,  northeast) 

Coe f f  (north,  east ) 

Coe f f  (north,  no-edge) 

Coeff  (northeast ,  northwest) 


0.50 

1.00 

0.50 

-0.50 

-0.25 

-0.50 

-0.75 

-0.50 

0.50 

-0.75 

0.75 


-0.50 

0.25 

0.25 

-0.25 

0.50 

-0.5 


(The  other  required  compatibility  coefficients  can  be  derived  from  these  by  using 
symmetry  considerations!)  We  also  tried  variations  of  the  above,  e.g.,  reversing  the 
signs  of  some  coefficients;  there  were  no  significant  changes  in  results.  The  lesson 
seems  to  be  that  manual  selection  of  coefficients  is  exceedingly  difficult.  A  better 
policy  is  to  generate  coefficients  using  hcampat  or  pcampal  and  to  modify  the  values 
only  slightly. 


6.3.  Anomaly  Detection 

Anomaly  detection  is  a  two-label  problem;  each  pixel  is  part  either  of  the  background 
or  of  an  anomaly.  We  assume  that  the  background  comprises  the  majority  of  the 
image  and  that  it  can  be  modeled  as  a  constant  gray  level  plus  Gaussian  noise. 
Anomalies  are  regions  that  diverge  significantly  from  this  model.  The  relaxation  pro¬ 
cess  should  strengthen  the  probability  of  an  anomaly  label  for  dark  or  light  groups  of 
pixels  while  suppressing  the  anomaly  label  for  isolated  pixels  that  are  equally  bright. 

The  background  is  modeled  by  the  mean  gray  level  and  standard  deviation  for  the 
whole  image.  Then,  for  each  pixel  in  turn,  we  calculate  the  deviation  of  the  pixel 
value  from  the  background  value  (i.e.,  from  the  image  mean).  The  probability  that 
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the  pixel  is  a  background  pixel  is  then  taken  to  be  the  probability  that  a  deviation 
this  large  or  larger  can  occur  by  chance.  The  probability  that  the  pixel  is  part  of  an 
anomaly  is  taken  to  be  1.0  minus  this  value. 

It  is  somewhat  easier  to  present  the  mathematical  formulas  in  the  reverse  order.  Let 
us  suppose  that  a  pixel  is  t  standard  deviations  from  the  image  mean.  We  calculate 
the  probability  of  the  label  anomaly  as 


Prab  (background)  =  1—Prob  {anomaly). 


These  initial  probabilities  are  used  to  calculate  the  compatibility  coefficients  by 
means  of  the  hcompal  or  pcompal  routine.  They  also  serve  as  input  to  the  relaxation 
process.  After  a  number  of  relaxation  iterations,  the  pixels  for  which  the  anomaly 
label  has  the  higher  probability  are  displayed  as  white  points  in  a  binary  image. 

Figures  14-16  are  the  results  of  our  efforts  at  anomaly  detection.  Each  figure  is  a 
pair  of  images  that  document  the  results  for  one  example.  The  four  pictures  in  the 
top  image,  from  left  to  right,  are  the 

*  Original  scene. 

*  Anomaly  probabilities  mapped  to  gray  levels  0  to  255. 

*  Binary  image  formed  by  inverse  mapping  the  two-label  probability  .image 
with  the  prbimg  routine, 

*  Binary  result  of  one  iteration  of  relaxation. 

The  bottom  image  shows,  from  left  to  right,  the  results  of  2  through  5  iterations  of 
relaxation. 

Figure  14  begins  with  the  original  gray-level  image  of  a  composite  road  scene.  Figure 
15  is  similar,  but  derived  from  a  version  of  the  road  scene  that  has  been  "cleaned”  to 
remove  the  background  road  profile  and  make  the  vehicles  (anomalies)  more  dis¬ 
tinct.  (This  technique  is  part  of  the  SRI  road  tracker  [Quam7B].)  Figure  16  is  the 
Peleg  method  applied  to  the  cleaned  road  scene,  In  each  case,  relaxation  enhances 
the  background  noise  structure  until  it  swamps  the  anomaly  signal, 

The  lesson  from  this  sequence  is  that  a  good  initial  image  does  not  guarantee  a  good 
result,  The  compatibility  coefficients  derived  by  hcampat  or  pcompal  do  not  neces¬ 
sarily  encode  our  own  notions  of  image  "structure.''  Relaxation-based  anomaly  detec¬ 
tion  may  be  feasible,  but  not  by  using  the  straight-forward  approach  attempted  here. 


6.4.  Summary 

For  any.  task  and  type.. of  imagery,  .. the  user  must  choose  the 

*  Image-to-probability  and  inverse  mappings 

*  Size  (and  shape)  of  the  compatibility  neighborhood 

*  Method  of  computing  compatibility  coefficients 
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*  Relaxation  scheme,  HZR  or  Peleg 

*  Number  of  relaxation  iterations. 

We  can  now  give  some  guidance  on  these  matters  as  well  as  on  the  question  of 
whether  to  use  relaxation  at  all. 

We  have  presented  some  simple  image-to-prob ability  and  inverse  mappings.  Each 
mapping  is  designed  for  a  specific  application,  and  the  success  of  relaxation  process¬ 
ing  depends  critically  on  the  quality  of  the  initial  mapping.  While  we  can  say  little 
about  the  initial  imagery  or  the  mapping  function,  we  can  suggest  some  constraints 
on  their  combination— the  probability  image. 

The  probability  image  is  the  source  of  both  the  compatibility  coefficients  and  the 
relaxation  output.  Any  repetitive  structure  in  this  image  will  be  reflected  in  the 
coefficients  and  enhanced  in  the  output  image.  It  is  therefore  essential  that  the  pro¬ 
bability  image  have  the  following  characteristics: 

*  The  signal,  or  desired  characteristics,  should  dominate  the  noise. 

*  The  signal  must  be  spatially  correlated  within  the  chosen  neighborhood; 
the  noise,  however,  should  be  uncorrelated. 

*  The  signal  must  contain  some  spatial  relationships  and  not  others;  the 
noise  should  contain  all  relationships  equally. 

Relaxation  will  be  successful  to  the  extent  that  these  conditions  are  met. 

The  user  should  specify  a  neighborhood  just  large  enough  to  guarantee  the  above 
conditions.  An  application  with  large  contiguous  regions,  such  as  land  use 
classification,  might  benefit  from  the  noise  immunity  of  a  large  neighborhood.  Gen¬ 
erally.  though,  a  3  x  3  neighborhood  is  the  best  choice.  If  the  signal  does  not  dom¬ 
inate  the  noise  locally,  it  is  unlikely  to  dominate  it  within  larger  neighborhoods.  If  it 
does  dominate  on  the  average,  relaxation  provides  a  mechanism  for  propagating  the 
signal  a  short  distance  into  those  regions  where  the  signal  is  weak.  Larger  neighbor¬ 
hoods  would  increase  penetration  distance,  but  at  the  cost  of  greatly  increasing  the 
computation  time. 

Different  compatibility  coefficients  are  needed  by  different  relaxation  methods,  but 
the  user  is  faced  with  similar  choices  in  computing  them  They  may  be  computed 
separately  for  each  image,  jointly  for  an  ensemble  of  images,  or  theoretically  for 
some  image  model.  The  same  set  of  constraints  applies:  the  sampled  or  modeled 
population  must  have  biased  second-order  statistics  rather  than  an  equiprobable 
selection  of  spatial  relationships.  This  typically  prohibits  training  on  an  ensemble  of 
randomly  oriented  images. 

The  RELAX  hcompai  and  pcompat  routines  look  for  combinations  of  neighboring 
labels  that  occur  more  (or  less)  frequently  than  that  expected  if  the  label  assign¬ 
ments  were  statistically  independent.  These  estimates  are  calculated  under  the 
assumption  that  the  image  is  a  homogeneous  data  set,  ignoring  the  fact  that  the 
structure  in  one  part  of  the  image  may  be  significantly  different  from  that  of  other 
parts.  It  has  been  suggested  that  this  disadvantage  may  be  alleviated  by  carrying 
out  the' calculation  over  windows  of  the  image  [Nagin82].  However,  the  problem  61 
estimating  coefficients  from  small  samples  worsens  as  the  sample  size  decreases.  If 
we  have  many  labels,  a  large  neighborhood,  and  a  small  sampling  area,  the  number  of 
samples  with  similar  pixel  configurations  will  be  small  and  the  estimated  compatibil¬ 
ity  coefficients  will  be  unreliable. 

An  alternative  method  for  obtaining  compatibility  coefficients  is  for  the  user  to 
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estimate  them.  For  many  labels  and  a  large  neighborhood,  manual  entry  of  the 
coefficients  is  a  daunting  task.  Moreover,  except  for  obvious  cases  of  compatibility,  it 
is  difficult  to  arrive  at  good  coefficient  values  by  guessing.  Figure  13  shows  the  hope¬ 
less  results  we  obtained  when  we  guessed  what  we  believed  to  be  reasonable  values 
for  the  coefficients.  A  major  difficulty  was  our  reluctance  to  allow  independent  label 
combinations.  Generally  we  assigned  values  that  forced  combinations  to  be  either 
compatible  or  incompatible  rather  than  independent.  If  manually  entered  values  of 
the  coefficients  are  to  be  used,  we  suggest  considering  those  the  package  can  provide 
as  a  guide  to  assigning  reasonable  values;  slight  modifications  may  then  be  beneficial. 

Which  relaxation  scheme  should  be  used?  It  does  not  seem  to  matter.  One  method 
sometimes  does  a  little  better  than  the  other,  but  both  work  or  fail  together.  The 
principal  distinction  we  observed  was  that  the  relaxation  smoothing  effect  could  be 
reduced  for  the  Peleg  method  by  using  a  larger  neighborhood. 

The  last  question  that  needs  to  be  answered  is  how  many  iterations  of  relaxation 
should  be  performed.  There  appear  to  be  two  aspects  to  the  relaxation  process.  The 
first  three  or  four  iterations  often  show  moderate  enhancement,  while  later  ones  are 
often  dominated  by  blurring.  Little  change  occurs  after  ten  iterations.  Only  by  look¬ 
ing  at  the  output  can  one  ascertain  the  optimum  hatting  point,  but  approximately 
four  iterations  seems  to  be  a  good  rule  of  thumb. 

Relaxation  is  essentially  an  enhancement  operator,  with  the  structure  to  be 
4  enhanced  derived  from  the  image  rather  than  from  any  model  of  either  the  imagery 
or  the  application.  What  is  the  cost  of  this  signal  enhancement?  One  iteration  of 
relaxation,  using  a  3x3  neighborhood  over  a  12B  x  128  image,  took  approximately 
three  CPU  minutes  on  a  VAX  ll/7B0-a  considerable  cost  if  image  smoothing  is  the 
desired  result.  These  costs  increase  linearly  with  the  number  of  image  pixels,  the 
number  of  pixels  in  the  neighborhood,  and  the  number  of  iterations. 
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Figure  1.  HZR  relaxation,  using  a  3x3  neighborhood  and  hcampat  compatibility 
,  coefficients.  See  text  for  explanation  of  the  figure  format.  Edge  linking  during  the 
first  iterations  is  supplanted  by  blurring  or  spreading. 


Figure  2.  Further  iterations  of  the  example  shown  in  Figure  1.  Iterations  10.  20 . 

100  are  shown.  These  additional  iterations  only  spread  the  edges. 
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Figure  3.  Peleg  relaxation  using  a  3x3  neighborhood  and  pcompat  compatibility 
,  coefficients.  Results  are  similar  to  those  obtained  using  the  HZR  scheme.  See  Figure 
1  for  comparison. 


Figure  4.  An  aerial  image  -with  HZR  relaxation  using  a  3  x  3  neighborhood  and  hcam.- 
pat  compatibility  coefficients.  Relaxation  extracts  structure  from  noisy  edge  data. 


36 


Evaluation 


C*s>  o 
(J>IJ>&I5j 


Figure  5.  HZR  relaxation  using  a  7  x  7  neighborhood  and  hcompat  compatibility 
,  coefficients.  The  use  of  a  larger  neighborhood  in  the  HZR  relaxation  scheme  gen¬ 
erally  increases  edge  spreading. 
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Figure  6.  Peleg  relaxation  using  a  7x7  neighborhood  and  pcompat  compatibility 
coefficients.  The  larger  neighborhood  reduces  the  edge  spread  instead  of  increasing 
it. 
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Figure  7.  A  printed-circuit  board  image  with.  HZR  relaxation  using  a  3  x  3  neighbor¬ 
hood  and  hcampat  compatibility  coefficients.  Both  this  example  and  Figure  1  have 
sharp  edges,  but  this  example  lacks  bias  in  the  edge  orientations. 


Figure  8.  A  noisy  tank  image  with  HZR  relaxation  using  a  3x3  neighborhood  and 
hcampat  compatibility  coefficients.  Although  the  edge-detection  method  is  inap¬ 
propriate  here,  it  works  quite  well  for  smoothing  or  object  detection. 
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Figure  9.  HZR  relaxation  using  a  3x3  neighborhood  and  hcompat  compatibility 
.  coefficients  rounded  to  one  decimal  place.  Comparison  with  Figure  1  shows  that 
exact  values  of  the  coefficients  are  not  important. 
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Figure  10.  HZR  relaxation  using  a  3x3  neighborhood  and  hcompat  compatibility 
coefficients  that  are  twice  those  for  Figure  9.  Only  the  relative  values  of  the  compati¬ 
bility  coefficients  are  important. 
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Figure  11.  HZR  relaxation  using  a  3x3  HZR  and  compatibility  coefficients  derived 
«  from  the  aerial  scene  in  Figure  12.  Useful  compatibilities  may  be  derived  from  dis¬ 
similar  imagery. 


Figure  12.  HZR  relaxation  using  a  3  x  3  neighborhood  and  compatibility  coefficients 
derived  from  the  chair  scene  in  Figure  11.  Useful  compatibilities  may  be  derived 
from  dissimilar  imagery. 
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Figure  13.  HZR  relaxation  using  a  3  x  3  neighborhood  and  manually  selected  compa- 
•  tibility  coefficients;  see  text  for  details.  Although  these  coefficients  were  intuitively 
determined,  they  perform  very  badly. 
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Figure  14.  Anomaly  detection  by  HZR  relaxation  using  a  3x3  neighborhood  and 
hcompat  compatibility  coefficients.  Top  image:  the  original  scene,  the  anomaly  pro¬ 
babilities,  and  binary  outputs  of  iterations  0  and  1.  Bottom  image:  iterations  2 
through  5.  Spatially  correlated  noise  is  enhanced  along  with  the  signal. 
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Figure  15.  Anomaly  detection  by  HZR  relaxation  using  a  3x3  neighborhood  and 
hcompat  compatibility  coefficients.  These  results  are  obtained  from  a  "cleaned”  ver¬ 
sion  of  the  original  scene  used  in  Figure  14. 
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Figure  16.  Anomaly  detection  by  Peleg  relaxation  using  a  3x3  neighborhood  and 
pcompat  compatibility  coefficients.  Comparison  with  Figure  15  shows  that  HZR  and 
Peleg  methods  produce  similar  results. 
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Suggested  Improvements 


The  process  of  evaluation  has  turned  up  several  ways  to  improve  or  extend  the  current 
RELAX  implementation.  Comments  about  existing  features  have  been  made  at  the 
appropriate  junctures  throughout  this  document.  The  following  are  additional  sugges¬ 
tions  for  substantial  modifications  or  needed  research. 


*  Improved.  Coefficient  Entry  and  Editing 

Manual  entry  of  compatibility  coefficients  is  currently  very  awkward,  and 
once  entered  the  coefficients  cannot  be  displayed  or  altered.  The  Testbed 
view  program  was  developed  to  display  files  of  coefficients,  but  a  more  flexi¬ 
ble  display-and-editing  capability  is  needed  within  the  relaxation  package 
itself, 

One  way  to  reduce  the  burden  on  the  user  is  to  vise  symmetry  or  other  con¬ 
straints  to  reduce  the  number  of  coefficients  that  must  be  typed  in  Another 
is  to  allow  entry  of  important  individual  coefficients,  with  all  others  default¬ 
ing  to  the  central  value  (0.0  or  1.0).  (This  approach  could  be  extended  to  the 
relaxation  updating  formula,  with  only  important  terms  actually  being 
entered  into  the  computation.)  In  any  case,  a  coefficient  query  and  correc¬ 
tion  capability  would  be  very  useful. 


*  Ensemble  Coefficient  Extraction 

The  current  hcompat  and  pcompat  routines  extract  compatibility 
coefficients  from  one  probability  image.  There  may  be  applications  for  which 
average  coefficients  from  an  ensemble  of  similar  images  are  desired.  A  sim¬ 
ple  program  could  be  written  to  extract  coefficients  from  such  an  ensemble, 
or  to  combine  coefficient  matrices  derived  from  individual  images. 


•  Supervised  Learning 

Relaxation  enhancement  is  often  unpredictable  when  the  compatibility 
coefficients  are  derived  from  noisy  images.  One  could  argue  that  “cleaned" 
or  “ground  truth"  images  should  be  used  for  deriving  the  coefficients.  This 
would  ‘build'  signal  statistics'  into  ‘the  compatibility  coefficients-  directly 
instead  of  depending  on  desired  signals  to  dominate  the  noise  in  the  initial 
probability  image.  The  result  should  be  faster  convergence  and  better  signal 
enhancement  [Peleg7Ba,  EklundhBO], 
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*  Adaptive  Coefficients 

Since  relaxation  can  be  used  to  enhance  signals  and  suppress  noise,  it  may 
be  useful  for  producing  cleaned  images  for  the  estimation  of  compatibility 
coefficients.  In  the  limit,  new  coefficients  could  be  extracted  during  (or 
after)  each  application  of  relaxation  updating.  This  would  either  produce 
faster  enhancement  or  faster  degradation  of  the  image. 


*  Use  of  Decision  Logic 

The  Hummel-Zucker-Rosenfeld  or  Peleg  updating  formulas  in  the  RELAX 
package  may  be  well  suited  for  enhancement  and  smoothing  applications. 
Many  other  uses  of  iterative  image  modification  would  require  nonlinear 
decision  logic  in  the  updating  algorithms.  The  decision  logic  might  make  use 
of  the  image  histogram,  the  statistics  of  objects  already  found  in  the  image, 
or  other  global  information 


*  £7se  0/  Joint  Neighborhood  Constraints 

The  current  probability  updating  formulas  use  only  pairwise  relationships  to 
compute  a  new  pixel  label  probability.  Other  types  of  iterative  image  opera¬ 
tors  often  look  for  patterns  in  the  neighborhood  as  a  whole.  There  may  be 
relaxation  applications  for  which  joint  neighborhood  relationships  must  simi¬ 
larly  be  modeled.  See  Peleg  [PelegBOa]  for  a  discussion  of  the  conditional 
independence  assumption. 


*  Adaptive  Neighborhood  Definition 

A  particular  use  of  joint  neighborhood  constraints  and  decision  logic  is  to 
decide,  for  each  neighborhood,  which  pixels  belong  to  the  same  region  as  the 
central  pixel.  Only  those  pixels  would  be  used  in  updating  the  central-pixel 
label  probabilities.  This  should  speed  convergence  in  segmentation  applica¬ 
tions. 


•  Halting  Criteria 

The  RELAX  package  currently  offers  no  way  to  ascertain  how  many  iterations 
of  relaxation  updating  are  sufficient  for  any  given  task.  We  have  suggested 
that  three  or  four  iterations  are  usually  optimum  for  enhancement  applica¬ 
tions,  but  there  are  no  image-dependent  rules  for  determining  when 
improvement  has  stopped  and  blurring  has  taken  over.  More  research  is 
necessary  in  this  area.  See  Fekete  ef  a l.  [FeketeBl]  for  an  approach  based 
on  examining  the  rates  of  change  and  the  entropies  of  the  probability  vec¬ 
tors  at  each  iteration. 
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*  Further  Research 

We  have  attempted  to  evaluate  relaxation  as  a  technique  rather  than  make 
an  exhaustive  study  of  its  application  to  a  particular  task.  If  relaxation 
seems  promising  for  a  specific  task,  however,  such  a  thorough  evaluation 
may  be  required.  As  relaxation  techniques  are  still  in  their  infancy,  further 
research  is  needed  to  determine  where  and  how  they  may  be  best  applied. 
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Conclusions 


Relaxation  is  a  procedure  for  enhancing  the  signal,  or  features,  found  in  an  image  by  an 
imperfect  enhancement,  detection,  or  classification  operator.  It  is  a  very  general  tech¬ 
nique  and  has  been  used  in  a  variety  of  image-processing  applications. 

The  approach  works  when  a  label  at  one  image  pixel  is  constrained  by  neighboring 
labels.  The  relaxation  procedure  discovers  and  exploits  these  relationships  to  produce 
a  more  consistent  labeling.  Where  an  initial  label  is  strongly  believed,  it  tends  to  be 
unchanged  by  relaxation  updating.  Where  it  is  uncertain,  relaxation  tends  to  pro¬ 
pagate  either  the  neighborhood  information  or  its  own  biases  into  the  classification. 
This  results  in  either  enhancement  or  smoothing,  depending  on  the  nature  of  the  com¬ 
patibility  coefficients. 

There  are  three  basic  components  to  relaxation:  mapping  the  original  image  to  a  pro¬ 
bability  domain,  estimating  the  compatibility  coefficients,  and  applying  the  updating 
formula.  The  updating  formulas  in  the  RELAX  package  are  simple,  standard,  and  nearly 
equivalent,  so  that  only  the  first  two  components  are  of  concern. 

Each  application  domain  requires  a  different  mapping  to  the  probability  image  format. 
The  mapping  should  be  such  that  (l)  the  desired  signal  dominates  other  image  com¬ 
ponents;  (2)  the  signal  is  locally  correlated  and  occurs  in  only  certain  neighboring 
combinations;  (3)  the  noise  is  locally  uncorrelated  and  appears  in  all  possible  combina¬ 
tions. 

Compatibility  coefficients  can  be  provided  by  hand,  although  they  are  exceedingly 
difficult  to  derive  for  most  applications.  The  automated  coefficient  extraction  routines 
in  the  RELAX  package  work  well  if  the  mapping  constraints  above  are  met,  but  produce 
surprising  results  otherwise.  If  the  noise  or  -unwanted  signal  in  an  image  is  spatially 
correlated,  it  will  be  enhanced.  If  the  desired  signal  takes  on  all  possible  local  relation¬ 
ships,  it  will  not  be  enhanced.  Enhancement  using  image-based  compatibility 
coefficients  can  improve  on  a  good  initial  image,  but  will  not  redeem  an  incompetent 
detection  operator. 

Relaxation  methods  for  solving  "gravitational”  or  "fluid  flow"  problems  such  as  histo¬ 
gram  sharpening,  requantization,  image  smoothing,  and  classification-map  improve¬ 
ment  have  been  reported  in  the  literature.  Relaxation  can  be  used  for  model- 
independent  enhancement,  but  is  often  more  costly  and  less  effective  than  model- 
based  enhancement  or  restoration  when  appropriate  models  are  available. 

The  RELAX  package  provides  a  mechanism  for  exploring  the  relaxation  philosophy  in 
image-based  applications.  Relaxation  techniques  are  still  in  an  early  stage  of  develop¬ 
ment,  and  more  research  is  needed  into  both  theoretical  foundations  and  domains  of 
applicability. 
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Appendix  A 


The  GPSFAR  Relaxation  Package 


This  appendix  documents  the  control  language  used  by  the  original  University  of  Mary¬ 
land  contribution  to  the  Testbed,  the  GPSPAR  relaxation  package.  This  is  a  set  of 
stand-alone  programs  that  may  be  invoked  in  sequence,  either  interactively  or  using  a 
UNIX  shell  script. 

The  capabilities  and  user  interfaces  of  the  GPSPAR  programs  are  essentially  identical 
to  those  documented  for  the  interactive  RELAX  driver.  (We  have  changed  the  names  of 
some  of  the  programs  during  integration  into  the  Testbed;  for  instance  prbimg  was  ori¬ 
ginally  known  as  display.  The  shell  script  is  just  another  method  of  invoking  these  pro¬ 
grams.  A  sample  script  is  shown  below. 

#  This  program,  will  do  everything  a  person  sitting 

#  at  a  terminal  would  do  to: 

§  Set  up  compat ih i 1 i ty  coefficient  creation  and 

f  relaxation  programs  using  the  Humne  1 -Zucker -Rosenf el d 

#  f  o  ram  las, 

# 

#  Create  a  two  label  probabilistic  image  from  the 

#  gray  level  "tank”  picture  (or  other  image)  using 

#  the  pr ob 1 em-  spec i f i c  program  "imgprb”, 

# 

#  Compute  the  compatibility  coefficients  from  this 

#  image  using  the  program  "hcompat”  that  was  produced 

#  by  "setup”. 

# 

#  Perform  eight  iterations  of  relaxation  using  "hrelax” 

#  as  well  bb  converting  each  resulting  probabilistic 

#  image  into  a  gray  level  image,  output. img,  using  the 

#  "prbimg”  program. 


#  Erase  the  screen, 
erase 


#  Sake  3x3  neighborhood,  two  label,  HZB  coefficient 

#  computation  and  relaxation  programs. 

esh  / in /t b/b in/ setup . esh  h  Z 

#  Transform  the  picture  into  a  probabilistic  image. 

§  SI  is  the  image  name,  SZ  and  S3  are  the  optional 

#  low  and  high  pixel  range  specifications. 

I 

if  (S#argv  <  1)  then 

imgprb  / iu/ tb /p i c / tank/bw.  img  prb.img  Z  13  49 
else 

imgprb  SI  prb.img  Z  SZ  S3 
endi  f 
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#  Recreate  a  gray  level  image  from  this.  This  first 

#  "output''  image  will  have  gone  through  essentially 

#  the  some  steps  as  the  other  new  gray  level  images 

#  to  be  created.  The  output  is  stretched  to  fill 

#  B -b i t  pi xe 1 s . 

prbimg  prb.img  outputO.img 
show  outputO.img  -t  100  348 


4  Compute  the  compatibility  coefficients  from  the 
4  probabilistic  image, 
hcompat  prb.img  compat.dat 


#  Perform  eight  iterations  of  relaxation  on  the  image. 
4  After  each  iteration  display  a  gray  level  image 
4  representation. 

hrelax  prb.img  compat.dat 
prbimg  prb.img  outputl.img 
show  outputl.img  - i  -t  224  348 

hrelax  prb.img  compat.dat 
prbimg  prb.img  outputs,  img 
show  outputs . img  -i  -t  348  348 

hrelax  prb.img  compat.dat 
prbimg  prb.img  outputs. img 
show  outputs. img  -i  -t  100  224 

hrelax  prb.img  compat.dat 
prbimg  prb.img  output4.  img 
show  output4. img  -i  -t  224  224 

hrelax  prb.img  compat.dat 
prbimg  prb.img  output5.img 
show  outputs. img  -i  -t  348  224 

hrelax  prb.img  compat.dat 
prbimg  prb.img  outputs. img 
show  outputs. img  -i  -t  100  100 

hrelax  prb.img  compat.dat 
prbimg  prb.img  output7.img 
show  output7.img  -i  -t  224  100 

hrelax  prb.img  compat.dat 
prbimg  prb.img  outputO.img 
show  outputs. img  -i  -t  348  100 

4  done 

echo  "Finished." 
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